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CONCLUSION 


This book is dedicated to the past and the future: 
In memory of my Beloved, Jim Maxmin. 
In memory of my courageous friend, Frank Schirrmacher. 
In honor of my children, 
Chloe Sophia Maxmin and Jacob Raphael Maxmin— 
I write to fortify your futures and the moral cause of your generation. 


THE DEFINITION 


Sur-veil-lance Cap-i-tal-ism, n. 


1. A new economic order that claims human experience as free raw material for hidden commercial 
practices of extraction, prediction, and sales; 2. A parasitic economic logic in which the production of 
goods and services is subordinated to a new global architecture of behavioral modification; 3. A rogue 
mutation of capitalism marked by concentrations of wealth, knowledge, and power unprecedented in 
human history; 4. The foundational framework of a surveillance economy; 5. As significant a threat to 
human nature in the twenty-first century as industrial capitalism was to the natural world in the nineteenth 
and twentieth; 6. The origin of a new instrumentarian power that asserts dominance over society and 
presents startling challenges to market democracy; 7. A movement that aims to impose a new collective 
order based on total certainty; 8. An expropriation of critical human rights that is best understood as a 
coup from above: an overthrow of the people’s sovereignty. 


INTRODUCTION 


CHAPTER ONE 


HOME OR EXILE IN THE DIGITAL FUTURE 


I saw him crying, shedding floods of tears upon 
Calypso’s island, in her chambers. 
She traps him there; he cannot go back home. 


—HOMER, THE ODYSSEY 


I. The Oldest Questions 


“Are we all going to be working for a smart machine, or will we have smart people around the machine?” 
The question was posed to me in 1981 by a young paper mill manager sometime between the fried catfish 
and the pecan pie on my first night in the small southern town that was home to his mammoth plant and 
would become my home periodically for the next six years. On that rainy night his words flooded my 
brain, drowning out the quickening tap tap tap of raindrops on the awning above our table. I recognized 
the oldest political questions: Home or exile? Lord or subject? Master or slave? These are eternal themes 
of knowledge, authority, and power that can never be settled for all time. There is no end of history; each 
generation must assert its will and imagination as new threats require us to retry the case in every age. 

Perhaps because there was no one else to ask, the plant manager’s voice was weighted with urgency 
and frustration: “What’s it gonna be? Which way are we supposed to go? I must know now. There is no 
time to spare.” I wanted the answers, too, and so I began the project that thirty years ago became my first 
book, In the Age of the Smart Machine: The Future of Work and Power. That work turned out to be the 
opening chapter in what became a lifelong quest to answer the question “Can the digital future be our 
home?” 

It has been many years since that warm southern evening, but the oldest questions have come roaring 
back with a vengeance. The digital realm is overtaking and redefining everything familiar even before we 
have had a chance to ponder and decide. We celebrate the networked world for the many ways in which it 
enriches our capabilities and prospects, but it has birthed whole new territories of anxiety, danger, and 
violence as the sense of a predictable future slips away. 

When we ask the oldest questions now, billions of people from every social strata, generation, and 
society must answer. Information and communications technologies are more widespread than electricity, 
reaching three billion of the world’s seven billion people! The entangled dilemmas of knowledge, 
authority, and power are no longer confined to workplaces as they were in the 1980s. Now their roots run 


deep through the necessities of daily life, mediating nearly every form of social participation.* 

Just a moment ago, it still seemed reasonable to focus our concerns on the challenges of an information 
workplace or an information society. Now the oldest questions must be addressed to the widest possible 
frame, which is best defined as “civilization” or, more specifically, information civilization. Will this 
emerging civilization be a place that we can call home? 

All creatures orient to home. It is the point of origin from which every species sets its bearings. 
Without our bearings, there is no way to navigate unknown territory; without our bearings, we are lost. I 
am reminded of this each spring when the same pair of loons returns from their distant travels to the cove 
below our window. Their haunting cries of homecoming, renewal, connection, and safeguard lull us to 
Sleep at night, knowing that we too are in our place. Green turtles hatch and go down to the sea, where 
they travel many thousands of miles, sometimes for ten years or twenty. When ready to lay their eggs, they 
retrace their journey back to the very patch of beach where they were born. Some birds annually fly for 
thousands of miles, losing as much as half their body weight, in order to mate in their birthplace. Birds, 
bees, butterflies... nests, holes, trees, lakes, hives, hills, shores, and hollows... nearly every creature 
shares some version of this deep attachment to a place in which life has been known to flourish, the kind 
of place we call home. 

It is in the nature of human attachment that every journey and expulsion sets into motion the search for 
home. That nostos, finding home, is among our most profound needs is evident by the price we are willing 
to pay for it. There is a universally shared ache to return to the place we left behind or to found a new 
home in which our hopes for the future can nest and grow. We still recount the travails of Odysseus and 
recall what human beings will endure for the sake of reaching our own shores and entering our own gates. 

Because our brains are larger than those of birds and sea turtles, we know that it is not always 
possible, or even desirable, to return to the same patch of earth. Home need not always correspond to a 
single dwelling or place. We can choose its form and location but not its meaning. Home is where we 
know and where we are known, where we love and are beloved. Home is mastery, voice, relationship, 
and sanctuary: part freedom, part flourishing... part refuge, part prospect. 

The sense of home slipping away provokes an unbearable yearning. The Portuguese have a name for 
this feeling: saudade, a word said to capture the homesickness and longing of separation from the 
homeland among emigrants across the centuries. Now the disruptions of the twenty-first century have 
turned these exquisite anxieties and longings of dislocation into a universal story that engulfs each one of 
us.2 


IT. Requiem for a Home 


In 2000 a group of computer scientists and engineers at Georgia Tech collaborated on a project called the 
“Aware Home.”4 It was meant to be a “living laboratory” for the study of “ubiquitous computing.” They 
imagined a “human-home symbiosis” in which many animate and inanimate processes would be captured 
by an elaborate network of “context aware sensors” embedded in the house and by wearable computers 
worn by the home’s occupants. The design called for an “automated wireless collaboration” between the 
platform that hosted personal information from the occupants’ wearables and a second one that hosted the 
environmental information from the sensors. 

There were three working assumptions: first, the scientists and engineers understood that the new data 
systems would produce an entirely new knowledge domain. Second, it was assumed that the rights to that 
new knowledge and the power to use it to improve one’s life would belong exclusively to the people who 


live in the house. Third, the team assumed that for all of its digital wizardry, the Aware Home would take 
its place as a modern incarnation of the ancient conventions that understand “home” as the private 
sanctuary of those who dwell within its walls. 

All of this was expressed in the engineering plan. It emphasized trust, simplicity, the sovereignty of the 
individual, and the inviolability of the home as a private domain. The Aware Home information system 
was imagined as a simple “closed loop” with only two nodes and controlled entirely by the home’s 
occupants. Because the house would be “constantly monitoring the occupants’ whereabouts and 
activities... even tracing its inhabitants’ medical conditions,” the team concluded, “there is a clear need 
to give the occupants knowledge and control of the distribution of this information.” All the information 
was to be stored on the occupants’ wearable computers “to insure the privacy of an individual’s 
information.” 

By 2018, the global “smart-home” market was valued at $36 billion and expected to reach $151 
billion by 2023.2 The numbers betray an earthquake beneath their surface. Consider just one smart-home 
device: the Nest thermostat, which was made by a company that was owned by Alphabet, the Google 
holding company, and then merged with Google in 2018.° The Nest thermostat does many things imagined 
in the Aware Home. It collects data about its uses and environment. It uses motion sensors and 
computation to “learn” the behaviors of a home’s inhabitants. Nest’s apps can gather data from other 
connected products such as cars, ovens, fitness trackers, and beds.2 Such systems can, for example, trigger 
lights if an anomalous motion is detected, signal video and audio recording, and even send notifications to 
homeowners or others. As a result of the merger with Google, the thermostat, like other Nest products, 
will be built with Google’s artificial intelligence capabilities, including its personal digital “assistant.”2 
Like the Aware Home, the thermostat and its brethren devices create immense new stores of knowledge 
and therefore new power—but for whom? 

Wi-Fi—enabled and networked, the thermostat’s intricate, personalized data stores are uploaded to 
Google’s servers. Each thermostat comes with a “privacy policy,” a “terms-of-service agreement,” and an 
“end-user licensing agreement.” These reveal oppressive privacy and security consequences in which 
sensitive household and personal information are shared with other smart devices, unnamed personnel, 
and third parties for the purposes of predictive analyses and sales to other unspecified parties. Nest takes 
little responsibility for the security of the information it collects and none for how the other companies in 
its ecosystem will put those data to use.2 A detailed analysis of Nest’s policies by two University of 
London scholars concluded that were one to enter into the Nest ecosystem of connected devices and apps, 
each with their own equally burdensome and audacious terms, the purchase of a single home thermostat 
would entail the need to review nearly a thousand so-called contracts. 

Should the customer refuse to agree to Nest’s stipulations, the terms of service indicate that the 
functionality and security of the thermostat will be deeply compromised, no longer supported by the 
necessary updates meant to ensure its reliability and safety. The consequences can range from frozen 
pipes to failed smoke alarms to an easily hacked internal home system. 

By 2018, the assumptions of the Aware Home were gone with the wind. Where did they go? What was 
that wind? The Aware Home, like many other visionary projects, imagined a digital future that empowers 
individuals to lead more-effective lives. What is most critical is that in the year 2000 this vision naturally 
assumed an unwavering commitment to the privacy of individual experience. Should an individual choose 
to render her experience digitally, then she would exercise exclusive rights to the knowledge garnered 
from such data, as well as exclusive rights to decide how such knowledge might be put to use. Today 
these rights to privacy, knowledge, and application have been usurped by a bold market venture powered 
by unilateral claims to others’ experience and the knowledge that flows from it. What does this sea change 


mean for us, for our children, for our democracies, and for the very possibility of a human future in a 
digital world? This book aims to answer these questions. It is about the darkening of the digital dream and 
its rapid mutation into a voracious and utterly novel commercial project that I call surveillance 
capitalism. 


III. What Is Surveillance Capitalism? 


Surveillance capitalism unilaterally claims human experience as free raw material for translation into 
behavioral data. Although some of these data are applied to product or service improvement, the rest are 
declared as a proprietary behavioral surplus, fed into advanced manufacturing processes known as 
“machine intelligence,” and fabricated into prediction products that anticipate what you will do now, 
soon, and later. Finally, these prediction products are traded in a new kind of marketplace for behavioral 
predictions that I call behavioral futures markets. Surveillance capitalists have grown immensely 
wealthy from these trading operations, for many companies are eager to lay bets on our future behavior. 

As we shall see in the coming chapters, the competitive dynamics of these new markets drive 
surveillance capitalists to acquire ever-more-predictive sources of behavioral surplus: our voices, 
personalities, and emotions. Eventually, surveillance capitalists discovered that the most-predictive 
behavioral data come from intervening in the state of play in order to nudge, coax, tune, and herd behavior 
toward profitable outcomes. Competitive pressures produced this shift, in which automated machine 
processes not only know our behavior but also shape our behavior at scale. With this reorientation from 
knowledge to power, it is no longer enough to automate information flows about us; the goal now is to 
automate us. In this phase of surveillance capitalism’s evolution, the means of production are 
subordinated to an increasingly complex and comprehensive “means of behavioral modification.” In this 
way, surveillance capitalism births a new species of power that I call instrumentarianism. 
Instrumentarian power knows and shapes human behavior toward others’ ends. Instead of armaments and 
armies, it works its will through the automated medium of an increasingly ubiquitous computational 
architecture of “smart” networked devices, things, and spaces. 

In the coming chapters we will follow the growth and dissemination of these operations and the 
instrumentarian power that sustains them. Indeed, it has become difficult to escape this bold market 
project, whose tentacles reach from the gentle herding of innocent Pokémon Go players to eat, drink, and 
purchase in the restaurants, bars, fast-food joints, and shops that pay to play in its behavioral futures 
markets to the ruthless expropriation of surplus from Facebook profiles for the purposes of shaping 
individual behavior, whether it’s buying pimple cream at 5:45 p.m. on Friday, clicking “yes” on an offer of 
new running shoes as the endorphins race through your brain after your long Sunday morning run, or 
voting next week. Just as industrial capitalism was driven to the continuous intensification of the means of 
production, so surveillance capitalists and their market players are now locked into the continuous 
intensification of the means of behavioral modification and the gathering might of instrumentarian power. 

Surveillance capitalism runs contrary to the early digital dream, consigning the Aware Home to 
ancient history. Instead, it strips away the illusion that the networked form has some kind of indigenous 
moral content, that being “connected” is somehow intrinsically pro-social, innately inclusive, or naturally 
tending toward the democratization of knowledge. Digital connection is now a means to others’ 
commercial ends. At its core, surveillance capitalism is parasitic and self-referential. It revives Karl 
Marx’s old image of capitalism as a vampire that feeds on labor, but with an unexpected turn. Instead of 
labor, surveillance capitalism feeds on every aspect of every human’s experience. 


Google invented and perfected surveillance capitalism in much the same way that a century ago 
General Motors invented and perfected managerial capitalism. Google was the pioneer of surveillance 
capitalism in thought and practice, the deep pocket for research and development, and the trailblazer in 
experimentation and implementation, but it is no longer the only actor on this path. Surveillance 
capitalism quickly spread to Facebook and later to Microsoft. Evidence suggests that Amazon has veered 
in this direction, and it is a constant challenge to Apple, both as an external threat and as a source of 
internal debate and conflict. 

As the pioneer of surveillance capitalism, Google launched an unprecedented market operation into 
the unmapped spaces of the internet, where it faced few impediments from law or competitors, like an 
invasive species ina landscape free of natural predators. Its leaders drove the systemic coherence of their 
businesses at a breakneck pace that neither public institutions nor individuals could follow. Google also 
benefited from historical events when a national security apparatus galvanized by the attacks of 9/11 was 
inclined to nurture, mimic, shelter, and appropriate surveillance capitalism’s emergent capabilities for the 
sake of total knowledge and its promise of certainty. 

Surveillance capitalists quickly realized that they could do anything they wanted, and they did. They 
dressed in the fashions of advocacy and emancipation, appealing to and exploiting contemporary 
anxieties, while the real action was hidden offstage. Theirs was an invisibility cloak woven in equal 
measure to the rhetoric of the empowering web, the ability to move swiftly, the confidence of vast revenue 
streams, and the wild, undefended nature of the territory they would conquer and claim. They were 
protected by the inherent illegibility of the automated processes that they rule, the ignorance that these 
processes breed, and the sense of inevitability that they foster. 

Surveillance capitalism is no longer confined to the competitive dramas of the large internet 
companies, where behavioral futures markets were first aimed at online advertising. Its mechanisms and 
economic imperatives have become the default model for most internet-based businesses. Eventually, 
competitive pressure drove expansion into the offline world, where the same foundational mechanisms 
that expropriate your online browsing, likes, and clicks are trained on your run in the park, breakfast 
conversation, or hunt for a parking space. Today’s prediction products are traded in behavioral futures 
markets that extend beyond targeted online ads to many other sectors, including insurance, retail, finance, 
and an ever-widening range of goods and services companies determined to participate in these new and 
profitable markets. Whether it’s a “smart” home device, what the insurance companies call “behavioral 
underwriting,” or any one of thousands of other transactions, we now pay for our own domination. 

Surveillance capitalism’s products and services are not the objects of a value exchange. They do not 
establish constructive producer-consumer reciprocities. Instead, they are the “hooks” that lure users into 
their extractive operations in which our personal experiences are scraped and packaged as the means to 
others’ ends. We are not surveillance capitalism’s “customers.” Although the saying tells us “If it’s free, 
then you are the product,” that is also incorrect. We are the sources of surveillance capitalism’s crucial 
surplus: the objects of a technologically advanced and increasingly inescapable raw-material-extraction 
operation. Surveillance capitalism’s actual customers are the enterprises that trade in its markets for 
future behavior. 

This logic turns ordinary life into the daily renewal of a twenty-first-century Faustian compact. 
“Faustian” because it is nearly impossible to tear ourselves away, despite the fact that what we must give 
in return will destroy life as we have known it. Consider that the internet has become essential for social 
participation, that the internet is now saturated with commerce, and that commerce is now subordinated to 
surveillance capitalism. Our dependency is at the heart of the commercial surveillance project, in which 
our felt needs for effective life vie against the inclination to resist its bold incursions. This conflict 


produces a psychic numbing that inures us to the realities of being tracked, parsed, mined, and modified. It 
disposes us to rationalize the situation in resigned cynicism, create excuses that operate like defense 
mechanisms (“I have nothing to hide”), or find other ways to stick our heads in the sand, choosing 
ignorance out of frustration and helplessness. In this way, surveillance capitalism imposes a 
fundamentally illegitimate choice that twenty-first-century individuals should not have to make, and its 
normalization leaves us singing in our chains. 

Surveillance capitalism operates through unprecedented asymmetries in knowledge and the power that 
accrues to knowledge. Surveillance capitalists know everything about us, whereas their operations are 
designed to be unknowable to us. They accumulate vast domains of new knowledge from us, but not for 
us. They predict our futures for the sake of others’ gain, not ours. As long as surveillance capitalism and 
its behavioral futures markets are allowed to thrive, ownership of the new means of behavioral 
modification eclipses ownership of the means of production as the fountainhead of capitalist wealth and 
power in the twenty-first century. 

These facts and their consequences for our individual lives, our societies, our democracies, and our 
emerging information civilization are examined in detail in the coming chapters. The evidence and 
reasoning employed here suggest that surveillance capitalism is a rogue force driven by novel economic 
imperatives that disregard social norms and nullify the elemental rights associated with individual 
autonomy that are essential to the very possibility of a democratic society. 

Just as industrial civilization flourished at the expense of nature and now threatens to cost us the Earth, 
an information civilization shaped by surveillance capitalism and its new instrumentarian power will 
thrive at the expense of human nature and will threaten to cost us our humanity. The industrial legacy of 
climate chaos fills us with dismay, remorse, and fear. As surveillance capitalism becomes the dominant 
form of information capitalism in our time, what fresh legacy of damage and regret will be mourned by 
future generations? By the time you read these words, the reach of this new form will have grown as more 
sectors, firms, startups, app developers, and investors mobilize around this one plausible version of 
information capitalism. This mobilization and the resistance it engenders will define a key battleground 
upon which the possibility of a human future at the new frontier of power will be contested. 


IV. The Unprecedented 


One explanation for surveillance capitalism’s many triumphs floats above them all: it is unprecedented. 
The unprecedented is necessarily unrecognizable. When we encounter something unprecedented, we 
automatically interpret it through the lenses of familiar categories, thereby rendering invisible precisely 
that which is unprecedented. A classic example is the notion of the “horseless carriage” to which people 
reverted when confronted with the unprecedented facts of the automobile. A tragic illustration is the 
encounter between indigenous people and the first Spanish conquerors. When the Tainos of the pre- 
Columbian Caribbean islands first laid eyes on the sweating, bearded Spanish soldiers trudging across 
the sand in their brocade and armor, how could they possibly have recognized the meaning and portent of 
that moment? Unable to imagine their own destruction, they reckoned that those strange creatures were 
gods and welcomed them with intricate rituals of hospitality. This is how the unprecedented reliably 
confounds understanding; existing lenses illuminate the familiar, thus obscuring the original by turning the 
unprecedented into an extension of the past. This contributes to the normalization of the abnormal, which 
makes fighting the unprecedented even more of an uphill climb. 

On a stormy night some years ago, our home was struck by lightning, and I learned a powerful lesson 


in the comprehension-defying power of the unprecedented. Within moments of the strike, thick black 
smoke drifted up the staircase from the lower level of the house and toward the living room. As we 
mobilized and called the fire department, I believed that I had just a minute or two to do something useful 
before rushing out to join my family. First, I ran upstairs and closed all the bedroom doors to protect them 
from smoke damage. Next, I tore back downstairs to the living room, where I gathered up as many of our 
family photo albums as I could carry and set them outside on a covered porch for safety. The smoke was 
just about to reach me when the fire marshal arrived to grab me by the shoulder and yank me out the door. 
We stood in the driving rain, where, to our astonishment, we watched the house explode in flames. 

I learned many things from the fire, but among the most important was the unrecognizability of the 
unprecedented. In that early phase of crisis, I could imagine our home scarred by smoke damage, but I 
could not imagine its disappearance. I grasped what was happening through the lens of past experience, 
envisioning a distressing but ultimately manageable detour that would lead back to the status quo. Unable 
to distinguish the unprecedented, all I could do was to close doors to rooms that would no longer exist 
and seek safety on a porch that was fated to vanish. I was blind to conditions that were unprecedented in 
my experience. 

I began to study the emergence of what I would eventually call surveillance capitalism in 2006, 
interviewing entrepreneurs and staff in a range of tech companies in the US and the UK. For several years 
I thought that the unexpected and disturbing practices that I documented were detours from the main road: 
management oversights or failures of judgment and contextual understanding. 

My field data were destroyed in the fire that night, and by the time I picked up the thread again early in 
2011, it was clear to me that my old horseless-carriage lenses could not explain or excuse what was 
taking shape. I had lost many details hidden in the brush, but the profiles of the trees stood out more 
clearly than before: information capitalism had taken a decisive turn toward a new logic of accumulation, 
with its own original operational mechanisms, economic imperatives, and markets. I could see that this 
new form had broken away from the norms and practices that define the history of capitalism and in that 
process something startling and unprecedented had emerged. 

Of course, the emergence of the unprecedented in economic history cannot be compared to a house 
fire. The portents of a catastrophic fire were unprecedented in my experience, but they were not original. 
In contrast, surveillance capitalism is a new actor in history, both original and sui generis. It is of its own 
kind and unlike anything else: a distinct new planet with its own physics of time and space, its sixty- 
seven-hour days, emerald sky, inverted mountain ranges, and dry water. 

Nonetheless, the danger of closing doors to rooms that will no longer exist is very real. The 
unprecedented nature of surveillance capitalism has enabled it to elude systematic contest because it 
cannot be adequately grasped with our existing concepts. We rely on categories such as “monopoly” or 
“privacy” to contest surveillance capitalist practices. And although these issues are vital, and even when 
surveillance capitalist operations are also monopolistic and a threat to privacy, the existing categories 
nevertheless fall short in identifying and contesting the most crucial and unprecedented facts of this new 
regime. 

Will surveillance capitalism continue on its current trajectory to become the dominant logic of 
accumulation of our age, or, in the fullness of time, will we judge it to have been a toothed bird: A 
fearsome but ultimately doomed dead end in capitalism’s longer journey? If it is to be doomed, then what 
will make it so? What will an effective vaccine entail? 

Every vaccine begins in careful knowledge of the enemy disease. This book is a journey to encounter 
what is strange, original, and even unimaginable in surveillance capitalism. It is animated by the 
conviction that fresh observation, analysis, and new naming are required if we are to grasp the 


unprecedented as a necessary prelude to effective contest. The chapters that follow will examine the 
specific conditions that allowed surveillance capitalism to root and flourish as well as the “laws of 
motion” that drive the action and expansion of this market form: its foundational mechanisms, economic 
imperatives, economies of supply, construction of power, and principles of social ordering. Let’s close 
doors, but let’s make sure that they are the right ones. 


V. The Puppet Master, Not the Puppet 


Our effort to confront the unprecedented begins with the recognition that we hunt the puppet master, not 
the puppet. A first challenge to comprehension is the confusion between surveillance capitalism and the 
technologies it employs. Surveillance capitalism is not technology; it is a logic that imbues technology 
and commands it into action. Surveillance capitalism is a market form that is unimaginable outside the 
digital milieu, but it is not the same as the “digital.” As we saw in the story of the Aware Home, and as 
we Shall see again in Chapter 2, the digital can take many forms depending upon the social and economic 
logics that bring it to life. It is capitalism that assigns the price tag of subjugation and helplessness, not the 
technology. 

That surveillance capitalism is a logic in action and not a technology is a vital point because 
surveillance capitalists want us to think that their practices are inevitable expressions of the technologies 
they employ. For example, in 2009 the public first became aware that Google maintains our search 
histories indefinitely: data that are available as raw-material supplies are also available to intelligence 
and law-enforcement agencies. When questioned about these practices, the corporation’s former CEO 
Eric Schmidt mused, “The reality is that search engines including Google do retain this information for 
some time.”/4 

In truth, search engines do not retain, but surveillance capitalism does. Schmidt’s statement is a classic 
of misdirection that bewilders the public by conflating commercial imperatives and technological 
necessity. It camouflages the concrete practices of surveillance capitalism and the specific choices that 
impel Google’s brand of search into action. Most significantly, it makes surveillance capitalism’s 
practices appear to be inevitable when they are actually meticulously calculated and lavishly funded 
means to self-dealing commercial ends. We will examine this notion of “inevitabilism’ in depth in 
Chapter _7. For now, suffice to say that despite all the futuristic sophistication of digital innovation, the 
message of the surveillance capitalist companies barely differs from the themes once glorified in the 
motto of the 1933 Chicago World’s Fair: “Science Finds—Industry Applies—Man Conforms.” 

In order to challenge such claims of technological inevitability, we must establish our bearings. We 
cannot evaluate the current trajectory of information civilization without a clear appreciation that 
technology is not and never can be a thing in itself, isolated from economics and society. This means that 
technological inevitability does not exist. Technologies are always economic means, not ends in 
themselves: in modern times, technology’s DNA comes already patterned by what the sociologist Max 
Weber called the “economic orientation.” 

Economic ends, Weber observed, are always intrinsic to technology’s development and deployment. 
“Economic action” determines objectives, whereas technology provides “appropriate means.” In Weber’s 
framing, “The fact that what is called the technological development of modern times has been so largely 
oriented economically to profit-making is one of the fundamental facts of the history of technology.”? In a 
modern capitalist society, technology was, is, and always will be an expression of the economic 
objectives that direct it into action. A worthwhile exercise would be to delete the word “technology” 


from our vocabularies in order to see how quickly capitalism’s objectives are exposed. 

Surveillance capitalism employs many technologies, but it cannot be equated with any technology. Its 
operations may employ platforms, but these operations are not the same as platforms. It employs machine 
intelligence, but it cannot be reduced to those machines. It produces and relies on algorithms, but it is not 
the same as algorithms. Surveillance capitalism’s unique economic imperatives are the puppet masters 
that hide behind the curtain orienting the machines and summoning them to action. These imperatives, to 
indulge another metaphor, are like the body’s soft tissues that cannot be seen in an X-ray but do the real 
work of binding muscle and bone. We are not alone in falling prey to the technology illusion. It is an 
enduring theme of social thought, as old as the Trojan horse. Despite this, each generation stumbles into 
the quicksand of forgetting that technology is an expression of other interests. In modern times this means 
the interests of capital, and in our time it is surveillance capital that commands the digital milieu and 
directs our trajectory toward the future. Our aim in this book is to discern the laws of surveillance 
capitalism that animate today’s exotic Trojan horses, returning us to age-old questions as they bear down 
on our lives, our societies, and our civilization. 

We have stood at this kind of precipice before. “We’ve stumbled along for a while, trying to run a new 
civilization in old ways, but we’ve got to start to make this world over.” It was 1912 when Thomas 
Edison laid out his vision for a new industrial civilization in a letter to Henry Ford. Edison worried that 
industrialism’s potential to serve the progress of humanity would be thwarted by the stubborn power of 
the robber barons and the monopolist economics that ruled their kingdoms. He decried the “wastefulness” 
and “cruelty” of US capitalism: “Our production, our factory laws, our charities, our relations between 
capital and labor, our distribution—all wrong, out of gear.” Both Edison and Ford understood that the 
modern industrial civilization for which they harbored such hope was careening toward a darkness 
marked by misery for the many and prosperity for the few. 

Most important for our conversation, Edison and Ford understood that the moral life of industrial 
civilization would be shaped by the practices of capitalism that rose to dominance in their time. They 
believed that America, and eventually the world, would have to fashion a new, more rational capitalism 
in order to avert a future of misery and conflict. Everything, as Edison suggested, would have to be 
reinvented: new technologies, yes, but these would have to reflect new ways of understanding and 
fulfilling people’s needs; a new economic model that could turn those new practices into profit; and a new 
social contract that could sustain it all. A new century had dawned, but the evolution of capitalism, like 
the churning of civilizations, did not obey the calendar or the clock. It was 1912, and still the nineteenth 
century refused to relinquish its claim on the twentieth. 

The same can be said of our time. As I write these words, we are nearing the end of the second decade 
of the twenty-first century, but the economic and social contests of the twentieth continue to tear us apart. 
These contests are the stage upon which surveillance capitalism made its debut and rose to stardom as the 
author of a new chapter in the long saga of capitalism’s evolution. This is the dramatic context to which 
we will turn in the opening pages of Part I: the place upon which we must stand in order to evaluate our 
subject in its rightful context. Surveillance capitalism is not an accident of overzealous technologists, but 
rather a rogue capitalism that learned to cunningly exploit its historical conditions to ensure and defend its 
success. 


VI. The Outline, Themes, and Sources of this Book 


This book is intended as an initial mapping of a terra incognita, a first foray that I hope will pave the way 


for more explorers. The effort to understand surveillance capitalism and its consequences has dictated a 
path of exploration that crosses many disciplines and historical periods. My aim has been to develop the 
concepts and frameworks that enable us to see the pattern in what have appeared to be disparate concepts, 
phenomena, and fragments of rhetoric and practice, as each new point on the map contributes to 
materializing the puppet master in flesh and bone. 

Many of the points on this map are necessarily drawn from fast-moving currents in turbulent times. In 
making sense of contemporary developments, my method has been to isolate the deeper pattern in the 
welter of technological detail and corporate rhetoric. The test of my efficacy will be in how well this map 
and its concepts illuminate the unprecedented and empower us with a more cogent and comprehensive 
understanding of the rapid flow of events that boil around us as surveillance capitalism pursues its long 
game of economic and social domination. 

The Age of Surveillance Capitalism has four parts. Each presents four to five chapters as well as a 
final chapter intended as a coda that reflects on and conceptualizes the meaning of what has gone before. 
Part I addresses the foundations of surveillance capitalism: its origins and early elaboration. We begin in 
Chapter 2 by setting the stage upon which surveillance capitalism made its debut and achieved success. 
This stage setting is important because I fear that we have contented ourselves for too long with 
superficial explanations of the rapid rise and general acceptance of the practices associated with 
surveillance capitalism. For example, we have credited notions such as “convenience” or the fact that 
many of its services are “free.” Instead, Chapter 2 explores the social conditions that summoned the 
digital into our everyday lives and enabled surveillance capitalism to root and flourish. I describe the 
“collision” between the centuries-old historical processes of individualization that shape our experience 
as self-determining individuals and the harsh social habitat produced by a decades-old regime of 
neoliberal market economics in which our sense of self-worth and needs for self-determination are 
routinely thwarted. The pain and frustration of this contradiction are the condition that sent us careening 
toward the internet for sustenance and ultimately bent us to surveillance capitalism’s draconian quid pro 
quo. 

Part I moves on to a close examination of surveillance capitalism’s invention and early elaboration at 
Google, beginning with the discovery and early development of what would become its foundational 
mechanisms, economic imperatives, and “laws of motion.” For all of Google’s technological prowess and 
computational talent, the real credit for its success goes to the radical social relations that the company 
declared as facts, beginning with its disregard for the boundaries of private human experience and the 
moral integrity of the autonomous individual. Instead, surveillance capitalists asserted their right to 
invade at will, usurping individual decision rights in favor of unilateral surveillance and the self- 
authorized extraction of human experience for others’ profit. These invasive claims were nurtured by the 
absence of law to impede their progress, the mutuality of interests between the fledgling surveillance 
capitalists and state intelligence agencies, and the tenacity with which the corporation defended its new 
territories. Eventually, Google codified a tactical playbook on the strength of which its surveillance 
capitalist operations were successfully institutionalized as the dominant form of information capitalism, 
drawing new competitors eager to participate in the race for surveillance revenues. On the strength of 
these achievements, Google and its expanding universe of competitors enjoy extraordinary new 
asymmetries of knowledge and power, unprecedented in the human story. I argue that the significance of 
these developments is best understood as the privatization of the division of learning in society, the 
critical axis of social order in the twenty-first century. 

Part II traces the migration of surveillance capitalism from the online environment to the real world, a 
consequence of the competition for prediction products that approximate certainty. Here we explore this 


new reality business, as all aspects of human experience are claimed as raw-material supplies and 
targeted for rendering into behavioral data. Much of this new work is accomplished under the banner of 
“personalization,” a camouflage for aggressive extraction operations that mine the intimate depths of 
everyday life. As competition intensifies, surveillance capitalists learn that extracting human experience 
is not enough. The most-predictive raw-material supplies come from intervening in our experience to 
shape our behavior in ways that favor surveillance capitalists’ commercial outcomes. New automated 
protocols are designed to influence and modify human behavior at scale as the means of production is 
subordinated to a new and more complex means of behavior modification. We see these new protocols at 
work in Facebook’s contagion experiments and the Google-incubated augmented reality “game” Pokémon 
Go. The evidence of our psychic numbing is that only a few decades ago US society denounced mass 
behavior-modification techniques as unacceptable threats to individual autonomy and the democratic 
order. Today the same practices meet little resistance or even discussion as they are routinely and 
pervasively deployed in the march toward surveillance revenues. Finally, I consider surveillance 
capitalism’s operations as a challenge to the elemental right to the future tense, which accounts for the 
individual’s ability to imagine, intend, promise, and construct a future. It is an essential condition of free 
will and, more poignantly, of the inner resources from which we draw the will to will. I ask and answer 
the question How did they get away with it? Part II ends with a meditation on our once and future history. 
If industrial capitalism dangerously disrupted nature, what havoc might surveillance capitalism wreak 
on human nature ? 

Part III examines the rise of instrumentarian power; its expression in a ubiquitous sensate, networked, 
computational infrastructure that I call Big Other; and the novel and deeply antidemocratic vision of 
society and social relations that these produce. I argue that instrumentarianism is an unprecedented 
species of power that has defied comprehension in part because it has been subjected to the “horseless- 
carriage” syndrome. Instrumentarian power has been viewed through the old lenses of totalitarianism, 
obscuring what is different and dangerous. Totalitarianism was a transformation of the state into a project 
of total possession. Instrumentarianism and its materialization in Big Other signal the transformation of the 
market into a project of total certainty, an undertaking that is unimaginable outside the digital milieu and 
the logic of surveillance capitalism. In naming and analyzing instrumentarian power, I explore its 
intellectual origins in early theoretical physics and its later expression in the work of the radical 
behaviorist B. F. Skinner. 

Part III follows surveillance capitalism into a second phase change. The first was the migration from 
the virtual to the real world. The second is a shift of focus from the real world to the social world, as 
society itself becomes the new object of extraction and control. Just as industrial society was imagined as 
a well-functioning machine, instrumentarian society is imagined as a human simulation of machine 
learning systems: a confluent hive mind in which each element learns and operates in concert with every 
other element. In the model of machine confluence, the “freedom” of each individual machine is 
subordinated to the knowledge of the system as a whole. Instrumentarian power aims to organize, herd, 
and tune society to achieve a similar social confluence, in which group pressure and computational 
certainty replace politics and democracy, extinguishing the felt reality and social function of an 
individualized existence. The youngest members of our societies already experience many of these 
destructive dynamics in their attachment to social media, the first global experiment in the human hive. I 
consider the implications of these developments for a second elemental right: the right to sanctuary. The 
human need for a space of inviolable refuge has persisted in civilized societies from ancient times but is 
now under attack as surveillance capital creates a world of “no exit” with profound implications for the 
human future at this new frontier of power. 


In the final chapter I conclude that surveillance capitalism departs from the history of market 
capitalism in surprising ways, demanding both unimpeded freedom and total knowledge, abandoning 
capitalism’s reciprocities with people and society, and imposing a totalizing collectivist vision of life in 
the hive, with surveillance capitalists and their data priesthood in charge of oversight and control. 
Surveillance capitalism and its rapidly accumulating instrumentarian power exceed the historical norms 
of capitalist ambitions, claiming dominion over human, societal, and political territories that range far 
beyond the conventional institutional terrain of the private firm or the market. As a result, surveillance 
capitalism is best described as a coup from above, not an overthrow of the state but rather an overthrow 
of the people’s sovereignty and a prominent force in the perilous drift toward democratic deconsolidation 
that now threatens Western liberal democracies. Only “we the people” can reverse this course, first by 
naming the unprecedented, then by mobilizing new forms of collaborative action: the crucial friction that 
reasserts the primacy of a flourishing human future as the foundation of our information civilization. If the 
digital future is to be our home, then it is we who must make it so. 

My methods combine those of a social scientist inclined toward theory, history, philosophy, and 
qualitative research with those of an essayist: an unusual but intentional approach. As an essayist, I 
occasionally draw upon my own experiences. I do this because the tendency toward psychic numbing is 
increased when we regard the critical issues examined here as just so many abstractions attached to 
technological and economic forces beyond our reach. We cannot fully reckon with the gravity of 
surveillance capitalism and its consequences unless we can trace the scars they carve into the flesh of our 
daily lives. 

As a social scientist, I have been drawn to earlier theorists who encountered the unprecedented in 
their time. Reading from this perspective, I developed a fresh appreciation for the intellectual courage 
and pioneering insights of classic texts, in which authors such as Durkheim, Marx, and Weber boldly 
theorized industrial capitalism and industrial society as it rapidly constructed itself in their midst during 
the nineteenth and early twentieth centuries. My work here has also been inspired by mid-twentieth- 
century thinkers such as Hannah Arendt, Theodor Adorno, Karl Polanyi, Jean-Paul Sartre, and Stanley 
Milgram, who struggled to name the unprecedented in their time as they faced the comprehension-defying 
phenomena of totalitarianism and labored to grasp their trail of consequence for the prospects of 
humanity. My work has also been deeply informed by the many insights of visionary scholars, technology 
critics, and committed investigative journalists who have done so much to illuminate key points on the 
map that emerges here. 

During the last seven years I have focused closely on the top surveillance capitalist firms and their 
growing ecosystems of customers, consultants, and competitors, all of it informed by the larger context of 
technology and data science that defines the Silicon Valley zeitgeist. This raises another important 
distinction. Just as surveillance capitalism is not the same as technology, this new logic of accumulation 
cannot be reduced to any single company or group of companies. The top five internet companies— 
Apple, Google, Amazon, Microsoft, and Facebook—are often regarded as a single entity with similar 
strategies and interests, but when it comes to surveillance capitalism, this is not the case. 

First, it is necessary to distinguish between capitalism and surveillance capitalism. As I discuss in 
more detail in Chapter 3, that line is defined in part by the purposes and methods of data collection. When 
a firm collects behavioral data with permission and solely as a means to product or service improvement, 
it is committing capitalism but not surveillance capitalism. Each of the top five tech companies practices 
capitalism, but they are not all pure surveillance capitalists, at least not now. 

For example, Apple has so far drawn a line, pledging to abstain from many of the practices that I 
locate in the surveillance capitalist regime. Its behavior in this regard is not perfect, the line is sometimes 


blurred, and Apple might well change or contradict its orientation. Amazon once prided itself on its 
customer alignment and the virtuous circle between data collection and service improvement. Both firms 
derive revenues from physical and digital products and therefore experience less financial pressure to 
chase surveillance revenues than the pure data companies. As we see in Chapter 9, however, Amazon 
appears to be migrating toward surveillance capitalism, with its new emphasis on “personalized” 
services and third-party revenues. 

Whether or not a corporation has fully migrated to surveillance capitalism says nothing about other 
vital issues raised by its operations, from monopolistic and anticompetitive practices in the case of 
Amazon to pricing, tax strategies, and employment policies at Apple. Nor are there any guarantees for the 
future. Time will tell if Apple succumbs to surveillance capitalism, holds the line, or perhaps even 
expands its ambitions to anchor an effective alternative trajectory to a human future aligned with the 
ideals of individual autonomy and the deepest values of a democratic society. 

One important implication of these distinctions is that even when our societies address capitalist 
harms produced by the tech companies, such as those related to monopoly or privacy, those actions do not 
ipso facto interrupt a firm’s commitment to and continued elaboration of surveillance capitalism. For 
example, calls to break up Google or Facebook on monopoly grounds could easily result in establishing 
multiple surveillance capitalist firms, though at a diminished scale, and thus clear the way for more 
surveillance capitalist competitors. Similarly, reducing Google and Facebook’s duopoly in online 
advertising does not reduce the reach of surveillance capitalism if online advertising market share is 
simply spread over five surveillance capitalist firms or fifty, instead of two. Throughout this book I focus 
on the unprecedented aspects of surveillance capitalist operations that must be contested and interrupted if 
this market form is to be contained and vanquished. 

My focus in these pages tends toward Google, Facebook, and Microsoft. The aim here is not a 
comprehensive critique of these companies as such. Instead, I view them as the petri dishes in which the 
DNA of surveillance capitalism is best examined. As I suggested earlier, my goal is to map a new logic 
and its operations, not a company or its technologies. I move across the boundaries of these and other 
companies in order to compile the insights that can flesh out the map, just as earlier observers moved 
across many examples to grasp the once-new logics of managerial capitalism and mass production. It is 
also the case that surveillance capitalism was invented in the United States: in Silicon Valley and at 
Google. This makes it an American invention, which, like mass production, became a global reality. For 
this reason, much of this text focuses on developments in the US, although the consequences of these 
developments belong to the world. 

In studying the surveillance capitalist practices of Google, Facebook, Microsoft, and other 
corporations, I have paid close attention to interviews, patents, earnings calls, speeches, conferences, 
videos, and company programs and policies. In addition, between 2012 and 2015 I interviewed 52 data 
scientists from 19 different companies with a combined 586 years of experience in high-technology 
corporations and startups, primarily in Silicon Valley. These interviews were conducted as I developed 
my “ground truth’ understanding of surveillance capitalism and its material infrastructure. Early on I 
approached a small number of highly respected data scientists, senior software developers, and 
specialists in the “internet of things.” My interview sample grew as scientists introduced me to their 
colleagues. The interviews, sometimes over many hours, were conducted with the promise of 
confidentiality and anonymity, but my gratitude toward them is personal, and I publicly declare it here. 

Finally, throughout this book you will read excerpts from W. H. Auden’s Sonnets from China, along 
with the entirety of Sonnet XVIII. This cycle of Auden’s poems is dear to me, a poignant exploration of 
humanity’s mythic history, the perennial struggle against violence and domination, and the transcendent 


power of the human spirit and its relentless claim on the future. 


CHAPTER TWO 


AUGUST 9, 2011: SETTING THE STAGE FOR 
SURVEILLANCE CAPITALISM 


The dangers and the punishments grew greater, 
And the way back by angels was defended 
Against the poet and the legislator. 


—W. H. AUDEN 
SONNETS FROM CHINA, II 


On August 9, 2011, three events separated by thousands of miles captured the bountiful prospects and 


gathering dangers of our emerging information civilization. First, Silicon Valley pioneer Apple promised 
a digital dream of new solutions to old economic and social problems, and finally surpassed Exxon Mobil 
as the world’s most highly capitalized corporation. Second, a fatal police shooting in London sparked 
extensive rioting across the city, engulfing the country in a wave of violent protests. A decade of 
explosive digital growth had failed to mitigate the punishing austerity of neoliberal economics and the 
extreme inequality that it produced. Too many people had come to feel excluded from the future, 
embracing rage and violence as their only remedies. Third, Spanish citizens asserted their rights to a 
human future when they challenged Google by demanding “the right to be forgotten.” This milestone 
alerted the world to how quickly the cherished dreams of a more just and democratic digital future were 
shading into nightmare, and it foreshadowed a global political contest over the fusion of digital 
capabilities and capitalist ambitions. We relive that August day every day as in some ancient fable, 
doomed to retrace this looping path until the soul of our information civilization is finally shaped by 
democratic action, private power, ignorance, or drift. 


I. The Apple Hack 


Apple thundered onto the music scene in the midst of a pitched battle between demand and supply. On one 
side were young people whose enthusiasm for Napster and other forms of music file sharing expressed a 
new quality of demand: consumption my way, what I want, when I want it, where I want it. On the other 
side were music-industry executives who chose to instill fear and to crush that demand by hunting down 
and prosecuting some of Napster’s most-ardent users. Apple bridged the divide with a commercially and 


legally viable solution that aligned the company with the changing needs of individuals while working 
with industry incumbents. Napster hacked the music industry, but Apple appeared to have hacked 
capitalism. 

It is easy to forget just how dramatic Apple’s hack really was. The company’s profits soared largely 
on the strength of its iPod/iTunes/iPhone sales. Bloomberg Businessweek described Wall Street analysts 
as “befuddled” by this mysterious Apple “miracle.” As one gushed, “We can’t even model out some of the 
possibilities... It’s like a religion.”! Even today the figures are staggering: three days after the launch of 
the Windows-compatible iTunes platform in October 2003, listeners downloaded a million copies of the 
free iTunes software and paid for a million songs, prompting Steve Jobs to announce, “In less than one 
week we’ve broken every record and become the largest online music company in the world.”? Within a 
month there were five million downloads, then ten million three months later, then twenty-five million 
three months after that. Four and a half years later, in January 2007, that number rose to two billion, and 
six years later, in 2013, it was 25 billion. In 2008 Apple surpassed Walmart as the world’s largest music 
retailer. iPod sales were similarly spectacular, exploding from 1 million units per month after the music 
store’s launch to 100 million less than four years later, when Apple subsumed the iPod’s functions in its 
revolutionary iPhone, which drove another step-function of growth. A 2017 study of stock market returns 
concluded that Apple had generated more profit for investors than any other US company in the previous 
century.? 

One hundred years before the iPod, mass production provided the gateway to a new era when it 
revealed a parallel universe of economic value hidden in new and still poorly understood mass 
consumers who wanted goods, but at a price they could afford. Henry Ford reduced the price of an 
automobile by 60 percent with a revolutionary industrial logic that combined high volume and low unit 
cost. He called it “mass production,” summarized in his famous maxim “You can have any color car you 
want so long as it’s black.” 

Later, GM’s Alfred Sloan expounded on that principle: “By the time we have a product to show them 
[consumers], we are necessarily committed to selling that product because of the tremendous investment 
involved in bringing it to market.”4 The music industry’s business model was built on telling its 
consumers what they would buy, just like Ford and Sloan. Executives invested in the production and 
distribution of CDs, and it was the CD that customers would have to purchase. 

Henry Ford was among the first to strike gold by tapping into the new mass consumption with the 
Model T. As in the case of the iPod, Ford’s Model T factory was pressed to meet the immediate 
explosion of demand. Mass production could be applied to anything, and it was. It changed the framework 
of production as it diffused throughout the economy and around the world, and it established the 
dominance of a new mass-production capitalism as the basis for wealth creation in the twentieth century. 

The iPod/iTunes innovations flipped this century-old industrial logic, leveraging the new capabilities 
of digital technologies to invert the consumption experience. Apple rewrote the relationship between 
listeners and their music with a distinct commercial logic that, while familiar to us now, was also 
experienced as revolutionary when first introduced. 

The Apple inversion depended on a few key elements. Digitalization made it possible to rescue 
valued assets—in this case, songs—from the institutional spaces in which they were trapped. The costly 
institutional procedures that Sloan had described were eliminated in favor of a direct route to listeners. In 
the case of the CD, for example, Apple bypassed the physical production of the product along with its 
packaging, inventory, storage, marketing, transportation, distribution, and physical retailing. The 
combination of the iTunes platform and the iPod device made it possible for listeners to continuously 
reconfigure their songs at will. No two iPods were the same, and an iPod one week was different from 


the same iPod another week, as listeners decided and re-decided the dynamic pattern. It was an 
excruciating development for the music industry and its satellites—retailers, marketers, etc.—but it was 
exactly what the new listeners wanted. 

How should we understand this success? Apple’s “miracle” is typically credited to its design and 
marketing genius. Consumers’ eagerness to have “what I want, when, where, and how I want it” is taken 
as evidence of the demand for “convenience” and sometimes even written off as narcissism or petulance. 
In my view, these explanations pale against the unprecedented magnitude of Apple’s accomplishments. We 
have contented ourselves for too long with superficial explanations of Apple’s unprecedented fusion of 
capitalism and the digital rather than digging deeper into the historical forces that summoned this new 
form to life. 

Just as Ford tapped into a new mass consumption, Apple was among the first to experience explosive 
commercial success by tapping into a new society of individuals and their demand for individualized 
consumption. The inversion implied a larger story of a commercial reformation in which the digital era 
finally offered the tools to shift the focus of consumption from the mass to the individual, liberating and 
reconfiguring capitalism’s operations and assets. It promised something utterly new, urgently necessary, 
and operationally impossible outside the networked spaces of the digital. Its implicit promise of an 
advocacy-oriented alignment with our new needs and values was a confirmation of our inner sense of 
dignity and worth, ratifying the feeling that we matter. In offering consumers respite from an institutional 
world that was indifferent to their individual needs, it opened the door to the possibility of a new rational 
capitalism able to reunite supply and demand by connecting us to what we really want in exactly the ways 
that we choose. 

As I shall argue in the coming chapters, the same historical conditions that sent the iPod on its wild 
ride summoned the emancipatory promise of the internet into our everyday lives as we sought remedies 
for inequality and exclusion. Of most significance for our story, these same conditions would provide 
important shelter for surveillance capitalism’s ability to root and flourish. More precisely, the Apple 
miracle and surveillance capitalism each owes its success to the destructive collision of two opposing 
historical forces. One vector belongs to the longer history of modernization and the centuries-long 
societal shift from the mass to the individual. The opposing vector belongs to the decades-long 
elaboration and implementation of the neoliberal economic paradigm: its political economics, its 
transformation of society, and especially its aim to reverse, subdue, impede, and even destroy the 
individual urge toward psychological self-determination and moral agency. The next sections briefly 
sketch the basic contours of this collision, establishing terms of reference that we will return to throughout 
the coming chapters as we explore surveillance capitalism’s rapid rise to dominance. 


II. The Two Modernities 


Capitalism evolves in response to the needs of people in a time and place. Henry Ford was clear on this 
point: “Mass production begins in the perception of a public need.”2 At a time when the Detroit 
automobile manufacturers were preoccupied with luxury vehicles, Ford stood alone in his recognition of 
a nation of newly modernizing individuals—farmers, wage earners, and shopkeepers—who had little and 
wanted much, but at a price they could afford. Their “demand” issued from the same conditions of 
existence that summoned Ford and his men as they discovered the transformational power of a new logic 
of standardized, high-volume, low-unit-cost production. Ford’s famous “five-dollar day” was emblematic 
of a systemic logic of reciprocity. In paying assembly-line workers higher wages than anyone had yet 


imagined, he recognized that the whole enterprise of mass production rested upon a thriving population of 
mass consumers. 

Although the market form and its bosses had many failings and produced many violent facts, its 
populations of newly modernizing individuals were valued as the necessary sources of customers and 
employees. It depended upon its communities in ways that would eventually lead to a range of 
institutionalized reciprocities. On the outside the drama of access to affordable goods and services was 
bound by democratic measures and methods of oversight that asserted and protected the rights and safety 
of workers and consumers. On the inside were durable employment systems, career ladders, and steady 
increases in wages and benefits.® Indeed, considered from the vantage point of the last forty years, during 
which this market form was systematically deconstructed, its reciprocity with the social order, however 
vexed and imperfect, appears to have been one of its most-salient features. 

The implication is that new market forms are most productive when they are shaped by an allegiance 
to the actual demands and mentalities of people. The great sociologist Emile Durkheim made this point at 
the dawn of the twentieth century, and his insight will be a touchstone for us throughout this book. 
Observing the dramatic upheavals of industrialization in his time—factories, specialization, the complex 
division of labor—Durkheim understood that although economists could describe these developments, 
they could not grasp their cause. He argued that these sweeping changes were “caused” by the changing 
needs of people and that economists were (and remain) systematically blind to these social facts: 


The division of labor appears to us otherwise than it does to economists. For them, it essentially consists in greater production. For us, 


this greater productivity is only a necessary consequence, a repercussion of the phenomenon. If we specialize, it is not to produce 


more, but it is to enable us to live in the new conditions of existence that have been made for usZ 


The sociologist identified the perennial human quest to live effectively in our “conditions of 
existence” as the invisible causal power that summons the division of labor, technologies, work 
organization, capitalism, and ultimately civilization itself. Each is forged in the same crucible of human 
need that is produced by what Durkheim called the always intensifying “violence of the struggle” for 
effective life: “If work becomes more divided,” it is because the “struggle for existence is more acute.”® 
The rationality of capitalism reflects this alignment, however imperfect, with the needs that people 
experience as they try to live their lives effectively, struggling with the conditions of existence that they 
encounter in their time and place. 

When we look through this lens, we can see that those eager customers for Ford’s incredible Model T 
and the new consumers of iPods and iPhones are expressions of the conditions of existence that 
characterized their era. In fact, each is the fruit of distinct phases of a centuries-long process known as 
“individualization” that is the human signature of the modern era. Ford’s mass consumers were members 
of what has been called the “first modernity,”? but the new conditions of the “second modermity” produced 
a new kind of individual for whom the Apple inversion, and the many digital innovations that followed, 
would become essential. This second modernity summoned the likes of Google and Facebook into our 
lives, and, in an unexpected twist, helped to enable the surveillance capitalism that would follow. 

What are these modernities and how do they matter to our story? The advent of the individual as the 
locus of moral agency and choice initially occurred in the West, where the conditions for this emergence 
first took hold. First let’s establish that the concept of “individualization” should not be confused with the 
neoliberal ideology of “individualism” that shifts all responsibility for success or failure to a mythical, 
atomized, isolated individual, doomed to a life of perpetual competition and disconnected from 
relationships, community, and society. Neither does it refer to the psychological process of 


“individuation” that is associated with the lifelong exploration of self-development. Instead, 
individualization is a consequence of long-term processes of modernization.12 

Until the last few minutes of human history, each life was foretold in blood and geography, sex and kin, 
rank and religion. I am my mother’s daughter. I am my father’s son. The sense of the human being as an 
individual emerged gradually over centuries, clawed from this ancient vise. Around two hundred years 
ago, we embarked upon the first modern road where life was no longer handed down one generation to 
the next according to the traditions of village and clan. This “first modernity” marks the time when life 
became “individualized” for great numbers of people as they separated from traditional norms, meanings, 
and rules. That meant each life became an open-ended reality to be discovered rather than a certainty to 
be enacted. Even where the traditional world remains intact for many people today, it can no longer be 
experienced as the only possible story. 

I often think about the courage of my great-grandparents. What mixture of sadness, terror, and 
exhilaration did they feel when in 1908, determined to escape the torments of the Cossacks in their tiny 
village outside of Kiev, they packed their five children, including my four-year-old grandfather Max, and 
all their belongings into a wagon and pointed the horses toward a steamer bound for America? Like 
millions of other pioneers of this first modernity, they escaped a still-feudal world and found themselves 
improvising a profoundly new kind of life. Max would later marry Sophie and build a family far from the 
rhythms of the villages that birthed them. The Spanish poet Antonio Machado captured the exhilaration 
and daring of these first-modernity individuals in his famous song: “Traveler, there is no road; the road is 
made as you go.” This is what “search” has meant: a journey of exploration and self-creation, not an 
instant swipe to already composed answers. 

Still, the new industrial society retained many of the hierarchical motifs of the older feudal world in 
its patterns of affiliation based on class, race, occupation, religion, ethnicity, sex, and the leviathans of 
mass society: its corporations, workplaces, unions, churches, political parties, civic groups, and school 
systems. This new world order of the mass and its bureaucratic logic of concentration, centralization, 
standardization, and administration still provided solid anchors, guidelines, and goals for each life. 

Compared to their parents and all the generations before, Sophie and Max had to make things up on 
their own, but not everything. Sophie knew she would raise the family. Max knew he would earn their 
living. You adapted to what the world had on offer, and you followed the rules. Nor did anyone ask your 
opinion or listen if you spoke. You were expected to do what you were supposed to do, and little by little 
you made your way. You raised a nice family, and eventually you’d have a house, car, washing machine, 
and refrigerator. Mass production pioneers like Henry Ford and Alfred Sloan had found a way to get you 
these things at a price you could afford. 

If there was anxiety, it reflected the necessity of living up to the requirements of one’s roles. One was 
expected to suppress any sense of self that spilled over the edges of the given social role, even at 
considerable psychic cost. Socialization and adaptation were the materials of a psychology and sociology 
that regarded the nuclear family as the “factory” for the “production of personalities” ready-made for 
conformity to the social norms of mass society./2 Those “factories” also produced a great deal of pain: the 
feminine mystique, closeted homosexuals, church-going atheists, and back-alley abortions. Eventually, 
though, they even produced people like you and me. 

When I set out on the open road, there were few answers, nothing to emulate, no compass to follow 
except for the values and dreams that I carried inside me. I was not alone; the road was filled with so 
many others on the same kind of journey. The first modernity birthed us, but we brought a new mentality to 
life: a “second modernity.”/2 What began as a modern migration from traditional lifeways bloomed into a 
new society of people born to a sense of psychological individuality, with its double-edged birthright of 


liberation and necessity. We experience both the right and the requirement to choose our own lives. No 
longer content to be anonymous members of the mass, we feel our entitlement to self-determination, an 
obvious truth to us that would have been an impossible act of hubris for Sophie and Max. This mentality 
is an extraordinary achievement of the human spirit, even as it can be a life sentence to uncertainty, 
anxiety, and stress. 

Since the second half of the twentieth century, the individualization story has taken this new turn 
toward a “second modernity.” Industrialization modernity and the practices of mass production capitalism 
at its core produced more wealth than had ever been imagined possible. Where democratic politics, 
distributional policies, access to education and health care, and strong civil society institutions 
complemented that wealth, a new “society of individuals” first began to emerge. Hundreds of millions of 
people gained access to experiences that had once been the preserve of a tiny elite: university education, 
travel, improved life expectancy, disposable income, rising standards of living, broad access to consumer 
goods, varied communication and information flows, and specialized, intellectually demanding work. 

The hierarchical social compact and mass society of the first modernity promised predictable 
rewards, but their very success was the knife that cut us loose and sent us tumbling onto the shores of the 
second modernity, propelling us toward more-intricate and richly patterned lives. Education and 
knowledge work increased mastery of language and thought, the tools with which we create personal 
meaning and form our own opinions. Communication, information, consumption, and travel stimulated 
individual self-consciousness and imaginative capabilities, informing perspectives, values, and attitudes 
in ways that could no longer be contained by predefined roles or group identity. Improved health and 
longer life spans provided the time for a self-life to deepen and mature, fortifying the legitimacy of 
personal identity over and against a priori social norms. 

Even when we revert to traditional roles, these are choices now rather than absolute truths imposed at 
birth. As the great clinician of identity, Erik Erikson, once described it, “The patient of today suffers most 
under the problem of what he should believe and who he should—or... might—be or become; while the 
patient of early psychoanalysis suffered most under inhibitions which prevented him from being what and 
who he thought he knew he was.”/4 This new mentality has been most pronounced in wealthier countries, 
but research shows significant pluralities of second-modernity individuals in nearly every region of the 
world. 

The first modernity suppressed the growth and expression of self in favor of collective solutions, but 
by the second modernity, the self is all we have. The new sense of psychological sovereignty broke upon 
the world long before the internet appeared to amplify its claims. We learn through trial and error how to 
stitch together our lives. Nothing is given. Everything must be reviewed, renegotiated, and reconstructed 
on the terms that make sense to us: family, religion, sex, gender, morality, marriage, community, love, 
nature, social connections, political participation, career, food... 

Indeed, it was this new mentality and its demands that summoned the internet and the burgeoning 
information apparatus into our everyday lives. The burdens of life without a fixed destiny turned us 
toward the empowering information-rich resources of the new digital milieu as it offered new ways to 
amplify our voices and forge our own chosen patterns of connection. So profound is this phenomenon that 
one can say without exaggeration that the individual as the author of his or her own life is the protagonist 
of our age, whether we experience this fact as emancipation or affliction. 

Western modernity had formed around a canon of principles and laws that confer inviolable individual 
rights and acknowledge the sanctity of each individual life? However, it was not until the second 
modernity that felt experience began to catch up with formal law. This felt truth has been expressed in 
new demands to make actual in everyday life what is already established in law.28 


In spite of its liberating potential, the second modernity was slated to become a hard place to live, and 
our conditions of existence today reflect this trouble. Some of the challenges of the second modernity 
arise from the inevitable costs associated with the creation and sustenance of one’s own life, but second- 
modernity instability is also the result of institutionalized shifts in economic and social policies and 
practices associated with the neoliberal paradigm and its rise to dominance. This far-reaching paradigm 
has been aimed at containing, rechanneling, and reversing the secular wave of second-modernity claims to 
self-determination and the habitats in which those claims can thrive. We live in this collision between a 
centuries-old story of modernization and a decades-old story of economic violence that thwarts our 
pursuit of effective life. 

There is a rich and compelling literature that documents this turning point in economic history, and my 
aim here is simply to call attention to some of the themes in this larger narrative that are vital to our 
understanding of the collision: the condition of existence that summoned both the Apple “miracle” and 
surveillance capitalism’s subsequent gestation and growth.12 


III. The Neoliberal Habitat 


The mid-1970s saw the postwar economic order under siege from stagnation, inflation, and sharply 
reduced growth, most markedly in the US and the UK. There were also new pressures on the political 
order as second-modernity individuals—especially students, young workers, African Americans, women, 
Latinos, and other marginalized groups—mobilized around demands for equal rights, voice, and 
participation. In the US the Vietnam War was a focal point of social unrest, and the corruption exposed by 
the Watergate scandal triggered public insistence on political reform. In the UK inflation had strained 
industrial relations beyond the breaking point. In both countries the specter of apparently intractable 
economic decay combined with vocal new demands on the democratic social compact produced 
confusion, anxiety, and desperation among elected officials ill-equipped to judge why once-reliable 
Keynesian policies had failed to reverse the course. 

Neoliberal economists had been waiting in the wings for this opportunity, and their ideas flowed into 
the “policy vacuum” that now bedeviled both governments.22 Led by the Austrian economist Friedrich 
Hayek, fresh from his 1974 Nobel Prize, and his American counterpart Milton Friedman, who received 
the Nobel two years later, they had honed their radical free-market economic theory, political ideology, 
and pragmatic agenda throughout the postwar period at the fringe of their profession, under the shadow of 
Keynesian domination, and now their time had come. 

The free-market creed originated in Europe as a sweeping defense against the threat of totalitarian and 
communist collectivist ideologies. It aimed to revive acceptance of a self-regulating market as a natural 
force of such complexity and perfection that it demanded radical freedom from all forms of state 
oversight. Hayek explained the necessity of absolute individual and collective submission to the exacting 
disciplines of the market as an unknowable “extended order” that supersedes the legitimate political 
authority vested in the state: “Modern economics explains how such an extended order... constitutes an 
information-gathering process... that no central planning agency, let alone any individual, could know as a 
whole, possess, or control....”22 Hayek and his ideological brethren insisted on a capitalism stripped 
down to its raw core, unimpeded by any other force and impervious to any external authority. Inequality of 
wealth and rights was accepted and even celebrated as a necessary feature of a successful market system 
and as a force for progress. Hayek’s ideology provided the intellectual superstructure and legitimation 
for a new theory of the firm that became another crucial antecedent to the surveillance capitalist 


corporation: its structure, moral content, and relationship to society. 

The new conception was operationalized by economists Michael Jensen and William Meckling. 
Leaning heavily on Hayek’s work, the two scholars took an ax to the pro-social principles of the 
twentieth-century corporation, an ax that became known as the “shareholder value movement.” In 1976 
Jensen and Meckling published a landmark article in which they reinterpreted the manager as a sort of 
parasite feeding off the host of ownership: unavoidable, perhaps, but nonetheless an obstacle to 
shareholder wealth. They boldly argued that structural disconnect between owners and managers “can 
result in the value of the firm being substantially lower than it otherwise could be.”“4 If managers 
suboptimized the value of the firm to its owners in favor of their own preferences and comfort, it was only 
rational for them to do so. The solution, these economists argued, was to assert the market’s signal of 
value, the share price, as the basis for a new incentive structure intended to finally and decisively align 
managerial behavior with owners’ interests. Managers who failed to bend to the ineffable signals of 
Hayek’s “extended order” would quickly become prey to the “barbarians at the gate” in a new and vicious 
hunt for unrealized market value. 

In the “crisis of democracy” zeitgeist, the neoliberal vision and its reversion to market metrics was 
deeply attractive to politicians and policy makers, both as the means to evade political ownership of 
tough economic choices and because it promised to impose a new kind of order where disorder was 
feared. The absolute authority of market forces would be enshrined as the ultimate source of imperative 
control, displacing democratic contest and deliberation with an ideology of atomized individuals 
sentenced to perpetual competition for scarce resources. The disciplines of competitive markets promised 
to quiet unruly individuals and even transform them back into subjects too preoccupied with survival to 
complain. 

As the old collectivist enemies had receded, new ones took their place: state regulation and oversight, 
social legislation and welfare policies, labor unions and the institutions of collective bargaining, and the 
principles of democratic politics. Indeed, all these were to be replaced by the market’s version of truth, 
and competition would be the solution to growth. The new aims would be achieved through supply-side 
reforms, including deregulation, privatization, and lower taxes. 

Thirty-five years before Hayek and Friedman’s ascendance, the great historian Karl Polanyi wrote 
eloquently on the rise of the market economy. Polanyi’s studies led him to conclude that the operations of 
a self-regulating market are profoundly destructive when allowed to run free of countervailing laws and 
policies. He described the double movement: “a network of measures and policies... integrated into 
powerful institutions designed to check the action of the market relative to labor, land, and money.”2° 

The double movement, Polanyi argued, supports the market form while tethering it to society: 
balancing, moderating, and mitigating its destructive excesses. Polanyi observed that such 
countermeasures emerged spontaneously in every European society during the second half of the 
nineteenth century. Each constructed legislative, regulatory, and institutional solutions to oversee 
contested new arenas such as workers’ compensation, factory inspection, municipal trading, public 
utilities, food safety, child labor, and public safety. 

In the US the double movement was achieved through decades of social contest that harnessed 
industrial production, however imperfectly, to society’s needs. It appeared in the trust busting, civil 
society, and legislative reforms of the Progressive Era. Later it was elaborated in the legislative, 
juridical, social, and tax initiatives of the New Deal and the institutionalization of Keynesian economics 
during the post-World War II era: labor market, tax, and social welfare policies that ultimately increased 
economic and social equality.*2 The double movement was further developed in the legislative initiatives 
of the Great Society, especially civil rights law and landmark environmental legislation. Many scholars 


credit such countermeasures with the success of market democracy in the US and Europe, a political 
economics that proved far more adaptive in its ability to produce reciprocities of demand and supply than 
either leftist theorists or even Polanyi had imagined, and by mid-century the large corporation appeared to 
be a deeply rooted and durable modern social institution. 

The double movement was scheduled for demolition under the neoliberal flag, and implementation 
began immediately. In 1976, the same year that Jensen and Meckling published their pathbreaking 
analysis, President Jimmy Carter initiated the first significant efforts to radically align the corporation 
with Wall Street’s market metrics, targeting the airline, transportation, and financial sectors with a bold 
program of deregulation. What began as a “ripple” turned into “a tidal wave that washed away controls 
from large segments of the economy in the last two decades of the twentieth century.”2? The 
implementation that began with Carter would define the Reagan and Thatcher eras, virtually every 
subsequent US presidency, and much of the rest of the world, as the new fiscal and social policies spread 
to Europe and other regions in varying degrees. 

Thus began the disaggregation and diminishment of the US public firm! The public corporation as a 
social institution was reinterpreted as a costly error, and its long-standing reciprocities with customers 
and employees were recast as destructive violations of market efficiency. Financial carrots and sticks 
persuaded executives to dismember and shrink their companies, and the logic of capitalism shifted from 
the profitable production of goods and services to increasingly exotic forms of financial speculation. The 
disciplines imposed by the new market operations stripped capitalism down to its raw core, and by 1989 
Jensen confidently proclaimed the “eclipse of the public corporation.”=2 

By the turn of the century, as the foundational mechanisms of surveillance capitalism were just 
beginning to take shape, “shareholder value maximization” was widely accepted as the “objective 
function” of the firm.2? These principles, culled from a once-extremist philosophy, were canonized as 
standard practice across commercial, financial, and legal domains.*4 By 2000, US public corporations 
employed fewer than half as many Americans as they did in 1970. In 2009 there were only half as many 
public firms as in 1997. The public corporation had become “unnecessary for production, unsuited for 
stable employment and the provision of social welfare services, and incapable of proving a reliable long- 
term return on investment.”2° In this process the cult of the “entrepreneur” would rise to near-mythic 
prominence as the perfect union of ownership and management, replacing the rich existential possibilities 
of the second modernity with a single glorified template of audacity, competitive cunning, dominance, and 
wealth. 


IV. The Instability of the Second Modernity 


On August 9, 2011, around the same time that cheers erupted in Apple’s conference room, 16,000 police 
officers flooded the streets of London, determined to quell “the most widespread and prolonged 
breakdown of order in London’s history since the Gordon riot of 1780.”2/ The rioting had begun four 
nights earlier when a peaceful vigil triggered by the police shooting of a young man suddenly turned 
violent. In the days that followed, the number of rioters swelled as looting and arson spread to twenty-two 
of London’s thirty-two boroughs and other major cities across Britain.2® Over four days of street action, 
thousands of people caused property damage of over $50 million, and 3,000 people were arrested. 

Even as Apple’s ascension appeared to ratify the claims of second-modernity individuals, the streets 
of London told the grim legacy of a three-decade experiment in economic growth through exclusion. One 
week after the rioting, an article by sociologist Saskia Sassen in the Daily Beast observed that “if there’s 


one underlying condition, it has to do with the unemployment and bitter poverty among people who desire 
to be part of the middle class and who are keenly aware of the sharp inequality between themselves and 
their country’s wealthy elite. These are in many ways social revolutions with a small ‘r,’ protests against 
social conditions that have become unbearable.”=2 

What were the social conditions that had become so unbearable? Many analysts agreed that the 
tragedy of Britain’s riots was set into motion by neoliberalism’s successful transformation of society: a 
program that was most comprehensively executed in the UK and the US. Indeed, research from the London 
School of Economics based on interviews with 270 people who had participated in the rioting reported 
on the predominant theme of inequality: “no job, no money.”*2 The terms of reference in nearly every 
study sound the same drumbeat: = of opportunity, lack of access to education, marginalization, 
deprivation, grievance, hopelessness.*! And although the London riots differed substantially from other 
protests that preceded and followed, most notably the Indignados movement that began with a large-scale 
public mobilization in Madrid in May 2011 and the Occupy movement that would emerge on September 
17 in Wall Street’s Zuccotti Park, they shared a point of origin in the themes of economic inequality and 
exclusion.“ 

The US, the UK, and most of Europe entered the second decade of the twenty-first century facing 
economic and social inequalities ae isaac than anything since the Gilded Age and comparable to 
some of the world’s poorest countries. Despite a decade of explosive digital growth that included the 
Apple miracle and the penetration of the internet into everyday life, dangerous social divisions suggested 
an even more stratified and antidemocratic future. “In the age of new consensus financial policy 
stabilization,” one US economist wrote, “the economy has witnessed the largest transfer of income to the 
top in history." A sobering 2016 report from the International Monetary Fund warned of instability, 
concluding that the global trends toward neoliberalism “have not delivered as expected.” Instead, 
inequality had significantly diminished “the level and the durability of growth” while increasing volatility 
and creating permanent vulnerability to economic crisis.® 

The quest for effective life had been driven to the breaking point under the aegis of market freedom. 
Two years after the North London riots, research in the UK showed that by 2013, poverty fueled by lack 
of education San unemployment already excluded nearly a third of the population from routine social 
participation.“ Another UK report concluded, “Workers on low and middle incomes are experiencing the 
biggest decline in their living standards since reliable records began in the mid-19th Century.”““ By 2015, 
austerity measures had eliminated 19 percent, or 18 billion pounds, from the budgets of local authorities, 
had forced an 8 percent cut in aa protection spending, and had caused 150,000 pensioners to no longer 
enjoy access to vital services.“® Buy 2014 nearly half of the US population lived in functional poverty, 
with the highest wage in the bottom half of earners at about $34,000. A 2012 US Department of 
Agriculture survey showed that close to 49 million people lived in “food-insecure” households. 

In Capital in the Twenty-First Century, the French economist Thomas Piketty integrated years of 
income data to derive a general law of accumulation: the rate of return on capital tends to exceed the rate 
of economic growth. This tendency, summarized as r > g, is a dynamic that produces ever-more-extreme 
income divergence and with it a range of antidemocratic social consequences long predicted as 
harbingers of an eventual crisis of capitalism. In this context, Piketty cites the ways in which financial 
elites use their be earnings to fund a cycle of political capture that protects their interests from 
political challenge.2! Indeed, a 2015 New York Times report concluded that 158 US families and their 
corporations provided almost half ($176 million) of all the money that was raised by both political 
parties in support of presidential candidates in 2016, primarily in support of “Republican candidates who 
have pledged to pare regulations, cut taxes... and shrink entitlements.”°2 Historians, investigative 


journalists, economists, and political scientists have analyzed the intricate facts of a turn toward 
oligarchy, shining a light on the systematic campaigns of public influence and political capture that helped 
drive and preserve an extreme free-market agenda at the expense of democracy. 

A précis of Piketty’s extensive research may be stated simply: capitalism should not be eaten raw. 
Capitalism, like sausage, is meant to be cooked by a democratic society and its institutions because raw 
capitalism is antisocial. As Piketty warns, “A market economy... if left to itself... contains powerful 
forces of divergence, which are potentially threatening to democratic societies and to the values of social 
justice on which they are based.”=4 Many scholars have taken to describing these new conditions as 
neofeudalism, marked by the consolidation of elite wealth and power far beyond the control of ordinary 
people and the mechanisms of democratic consent. Piketty calls it a return to “patrimonial capitalism,” a 
reversion to a premodern society in which one’s life chances depend upon inherited wealth rather than 
meritocratic achievement.22 

We now have the tools to grasp the collision in all of its destructive complexity: what is unbearable is 
that economic and social inequalities have reverted to the preindustrial “feudal” pattern but that we, 
the people, have not. We are not illiterate peasants, serfs, or slaves. Whether “middle class” or 
“marginalized,” we share the collective historical condition of individualized persons with complex 
social experiences and opinions. We are hundreds of millions or even billions of second-modernity 
people whom history has freed both from the once-immutable facts of a destiny told at birth and from the 
conditions of mass society. We know ourselves to be worthy of dignity and the opportunity to live an 
effective life. This is existential toothpaste that, once liberated, cannot be squeezed back into the tube. 
Like a detonation’s rippling sound waves of destruction, the reverberations of pain and anger that have 
come to define our era arise from this poisonous collision between inequality’s facts and inequality’s 
feelings.>/ 

Back in 2011, those 270 interviews of London participants in the riots also reflected the scars of this 
collision. “They expressed it in different ways,” the report concludes, “but at heart what the rioters talked 
about was a pervasive sense of injustice. For some, this was economic—the lack of a job, money, or 
opportunity. For others it was more broadly social, not just the absence of material things, but how they 
felt they were treated compared with others....” The “sense of being invisible” was “widespread.” As 
one woman explained, “The young these days need to be heard. It’s got to be justice for them.” And a 
young man reflected, “When no one cares about you you’re gonna eventually make them care, you’re 
gomna cause a disturbance.”=8 Other analyses cite “the denial of dignity” expressed in the wordless anger 
of the North London rampage.22 

When the Occupy movement erupted on another continent far from London’s beleaguered 
neighborhoods, it appeared to have little in common with the violent eruptions that August. The 99 percent 
that Occupy intended to represent is not marginalized; on the contrary, the very legitimacy of Occupy was 
its claim to supermajority status. Nevertheless, Occupy revealed a similar conflict between inequality’s 
facts and inequality’s feelings, expressed in a creatively individualized political culture that insisted on 
“direct democracy” and “horizontal leadership.”©2 Some analysts concluded that it was this conflict that 
ultimately crippled the movement, with its “inner core” of leaders unwilling to compromise their highly 
individualized approach in favor of the strategies and tactics required for a durable mass movement.®! 
However, one thing is certain: there were no serfs in Zuccotti Park. On the contrary, as one close observer 
of the movement ruminated, “What is different is that from the start very large sections of we, the people, 
proved to be wiser than our rulers. We saw further and proved to have better judgment, thus reversing the 
traditional legitimacy of our elite governance that those in charge know better than the unwashed.”® 

This is the existential contradiction of the second modernity that defines our conditions of existence: 


we want to exercise control over our own lives, but everywhere that control is thwarted. 
Individualization has sent each one of us on the prowl for the resources we need to ensure effective life, 
but at each turn we are forced to do battle with an economics and politics from whose vantage point we 
are but ciphers. We live in the knowledge that our lives have unique value, but we are treated as invisible. 
As the rewards of late-stage financial capitalism slip beyond our grasp, we are left to contemplate the 
future in a bewilderment that erupts into violence with increasing frequency. Our expectations of 
psychological self-determination are the grounds upon which our dreams unfold, so the losses we 
experience in the slow burn of rising inequality, exclusion, pervasive competition, and degrading 
stratification are not only economic. They slice us to the quick in dismay and bitterness because we know 
ourselves to be worthy of individual dignity and the right to a life on our own terms. 

The deepest contradiction of our time, the social philosopher Zygmunt Bauman wrote, is “the yawning 
gap between the right of self-assertion and the capacity to control the social settings which render such 
self-assertion feasible. It is from that abysmal gap that the most poisonous effluvia contaminating the lives 
of contemporary individuals emanate.” Any new chapter in the centuries-old story of human emancipation, 
he insisted, must begin here. Can the instability of the second modernity give way to a new synthesis: a 
third modernity that transcends the collision, offering a genuine path to a flourishing and effective life for 
the many, not just the few? What role will information capitalism play? 


V. A Third Modernity 


Apple once launched itself into that “abysmal gap,” and for a time it seemed that the company’s fusion of 
capitalism and the digital might set a new course toward a third modernity. The promise of an advocacy- 
oriented digital capitalism during the first decade of our century galvanized second-modernity populations 
around the world. New companies such as Google and Facebook appeared to bring the promise of the 
inversion to life in new domains of critical importance, rescuing information and people from the old 
institutional confines, enabling us to find what and whom we wanted, when and how we wanted to search 
or comnect. 

The Apple inversion implied trustworthy relationships of advocacy and reciprocity embedded in an 
alignment of commercial operations with consumers’ genuine interests. It held out the promise of a new 
digital market form that might transcend the collision: an early intimation of a third-modernity capitalism 
summoned by the self-determining aspirations of individuals and indigenous to the digital milieu. The 
opportunity for “my life, my way, at a price I can afford” was the human promise that quickly lodged at 
the very heart of the commercial digital project, from iPhones to one-click ordering to massive open 
online courses to on-demand services to hundreds of thousands of web-based enterprises, apps, and 
devices. 

There were missteps, shortfalls, and vulnerabilities, to be sure. The potential significance of Apple’s 
tacit new logic was never fully grasped, even by the company itself. Instead, the corporation produced a 
steady stream of contradictions that signaled business as usual. Apple was criticized for extractive 
pricing policies, offshoring jobs, exploiting its retail staff, abrogating responsibility for factory 
conditions, colluding to depress wages via illicit noncompete agreements in employee recruitment, 
institutionalized tax evasion, and a lack of environmental stewardship—just to name a few of the 
violations that seemed to negate the implicit social contract of its own unique logic. 

When it comes to genuine economic mutation, there is always a tension between the new features of 
the form and its mother ship. A combination of old and new is reconfigured in an unprecedented pattern. 


Occasionally, the elements of a mutation find the right environment in which to be “selected” for 
propagation. This is when the new form stands a chance of becoming fully institutionalized and 
establishes its unique migratory path toward the future. But it’s even more likely that potential mutations 
meet their fate in “transition failure,” drawn back by the gravitational pull of established practices. 

Was the Apple inversion a powerful new economic mutation running the gauntlet of trial and error on 
its way to fulfilling the needs of a new age, or was it a case of transition failure? In our enthusiasm and 
growing dependency on technology, we tended to forget that the same forces of capital from which we had 
fled in the “real” world were rapidly claiming ownership of the wider digital sphere. This left us 
vulnerable and caught unawares when the early promise of information capitalism took a darker turn. We 
celebrated the promise of “help is on the way” while troubling questions broke through the haze with 
increasing regularity, each one followed by a predictable eruption of dismay and anger. 

Why did Google’s Gmail, launched in 2004, scan private correspondence to generate advertising? As 
soon as the first Gmail user saw the first ad targeted to the content of her private correspondence, public 
reaction was swift. Many were repelled and outraged; others were confused. As Google chronicler 
Steven Levy put it, “By serving ads related to content, Google seemed almost to be reveling in the fact 
that users’ privacy was at the mercy of the policies and trustworthiness of the company that owned the 
servers. And since those ads made profits, Google was making it clear that it would exploit the 
situation.”™ 

In 2007 Facebook launched Beacon, touting it as “a new way to socially distribute information.” 
Beacon enabled Facebook advertisers to track users across the internet, disclosing users’ purchases to 
their personal networks without permission. Most people were outraged by the company’s audacity, both 
in tracking them online and in usurping their ability to control the disclosure of their own facts. Facebook 
founder Mark Zuckerberg shut the program down under duress, but by 2010 he declared that privacy was 
no longer a social norm and then congratulated himself for relaxing the company’s “privacy policies” to 
reflect this self-interested assertion of a new social condition. Zuckerberg had apparently never read 
user Jonathan Trenn’s rendering of his Beacon experience: 


I purchased a diamond engagement ring set from overstock in preparation for a New Year’s surprise for my girlfriend... Within 
hours, I received a shocking call from one of my best friends of surprise and “congratulations” for getting engaged.(!!!) Imagine my 
horror when I learned that overstock had published the details of my purchase (including a link to the item and its price) on my public 
Facebook newsfeed, as well as notifications to all of my friends. ALL OF MY FRIENDS, including my girlfriend, and all of her 
friends, etc.... ALL OF THIS WAS WITHOUT MY CONSENT OR KNOWLEDGE. I am totally distressed that my surprise was 
ruined, and what was meant to be something special and a lifetime memory for my girlfriend and I was destroyed by a totally 


underhanded and infuriating privacy invasion. I want to wring the neck of the folks at overstock and facebook who thought that this 


was a good idea. It sets a terrible precedent on the net, and I feel that it ruined a part of my life 96 


Among the many violations of advocacy expectations, ubiquitous “terms-of-service agreements” were 
among the most pernicious.®2 Legal experts call these “contracts of adhesion” because they impose take- 
it-or-leave-it conditions on users that stick to them whether they like it or not. Online “contracts” such as 
terms-of-service or terms-of-use agreements are also referred to as “click-wrap” because, as a great deal 
of research shows, most people get wrapped in these oppressive contract terms by simply clicking on the 
box that says “I agree” without ever reading the agreement.© In many cases, simply browsing a website 
obligates you to its terms-of-service agreement even if you don’t know it. Scholars point out that these 
digital documents are excessively long and complex in part to discourage users from actually reading the 
terms, safe in the knowledge that most courts have upheld the legitimacy of click-wrap agreements despite 


the obvious lack of meaningful consent.22 US Supreme Court Chief Justice John Roberts admitted that he 
“doesn’t read the computer fine print.”2 Adding insult to injury, terms of service can be altered 
unilaterally by the firm at any time, without specific user knowledge or consent, and the terms typically 
implicate other companies (partners, suppliers, marketers, advertising intermediaries, etc.) without stating 
or accepting responsibility for their terms of service. These “contracts” impose an unwinnable infinite 
regress upon the user that law professor Nancy Kim describes as “sadistic.” 

Legal scholar Margaret Radin observes the Alice-in-Wonderland quality of such “contracts.” Indeed, 
the sacred notions of “agreement” and “promise” so critical to the evolution of the institution of contract 
since Roman times have devolved to a “talismanic” signal “merely indicating that the firm deploying the 
boilerplate wants the recipient to be bound.” Radin calls this “private eminent domain,” a unilateral 
seizure of rights without consent. She regards such “contracts” as a moral and democratic “degradation” 
of the rule of law and the institution of contract, a perversion that restructures the rights of users granted 
through democratic processes, “substituting for them the system that the firm wishes to impose.... 
Recipients must enter a legal universe of the firm’s devising in order to engage in transactions with the 
firm.”2 

The digital milieu has been essential to these degradations. Kim points out that paper documents once 
imposed natural restraints on contracting behavior simply by virtue of their cost to produce, distribute, 
and archive. Paper contracts require a physical signature, limiting the burden a firm is likely to impose on 
a customer by requiring her to read multiple pages of fine print. Digital terms, in contrast, are 
“weightless.” They can be expanded, reproduced, distributed, and archived at no additional cost. Once 
firms understood that the courts were disposed to validate their click-wrap and browse-wrap agreements, 
there was nothing to stop them from expanding the reach of these degraded contracts “to extract from 
consumers additional benefits unrelated to the transaction.” This coincided with the discovery of 
behavioral surplus that we examine in Chapter 3, as terms-of-service agreements were extended to 
include baroque and perverse “privacy policies,” establishing another infinite regress of these terms of 
expropriation. Even the former Federal Trade Commission Chairperson Jon Leibowitz publicly stated, 
“We all agree that consumers don’t read privacy policies.”4 In 2008 two Carnegie Mellon professors 
calculated that a reasonable reading of all the privacy policies that one encounters in a year would 
require 76 full workdays at a national opportunity cost of $781 billion. The numbers are much higher 
today. Still, most users remain unaware of these “rapacious” terms that, as Kim puts it, allow firms “to 
acquire rights without bargaining and to stealthily establish and embed practices before users, and 
regulators, realize what has happened.” 

At first, it had seemed that the new internet companies had simply failed to grasp the moral, social, 
and institutional requirements of their own economic logic. But with each corporate transgression, it 
became more difficult to ignore the possibility that the pattern of violations signaled a feature, not a bug. 
Although the Apple miracle contained the seeds of economic reformation, it was poorly understood: a 
mystery even to itself. Long before the death of its legendary founder, Steve Jobs, its frequent abuses of 
user expectations raised questions about how well the corporation understood the deep structure and 
historic potential of its own creations. The dramatic success of Apple’s iPod and iTunes instilled internet 
users with a sense of optimism toward the new digital capitalism, but Apple never did seize the reins on 
developing the consistent, comprehensive social and institutional processes that would have elevated the 
iPod’s promise to an explicit market form, as Henry Ford and Alfred Sloan had once done. 

These developments reflect the simple truth that genuine economic reformation takes time and that the 
internet world, its investors and shareholders, were and are in a hurry. The credo of digital innovation 
quickly turned to the language of disruption and an obsession with speed, its campaigns conducted under 


the flag of “creative destruction.” That famous, fateful phrase coined by evolutionary economist Joseph 
Schumpeter was seized upon as a way to legitimate what Silicon Valley euphemistically calls 
“permissionless innovation.” Destruction rhetoric promoted what I think of as a “boys and their toys” 
theory of history, as if the winning hand in capitalism is about blowing things up with new technologies. 
Schumpeter’s analysis was, in fact, far more nuanced and complex than modern destruction rhetoric 
suggests. 

Although Schumpeter regarded capitalism as an “evolutionary” process, he also considered that 
relatively few of its continuous innovations actually rise to the level of evolutionary significance. These 
rare events are what he called “mutations.” These are enduring, sustainable, qualitative shifts in the logic, 
understanding, and practice of capitalist accumulation, not random, temporary, or opportunistic reactions 
to circumstances. Schumpeter insisted that this evolutionary mechanism is triggered by new consumer 
needs, and alignment with those needs is the discipline that drives sustainable mutation: “The capitalist 
process, not by coincidence but by virtue of its mechanism, progressively raises the standard of life of the 
masses.” 

If a mutation is to be reliably sustained, its new aims and practices must be translated into new 
institutional forms: “The fundamental impulse that sets and keeps the capitalist engine in motion comes 
from the new consumers’ goods, the new methods of production or transportation, the new markets, the 
new forms of industrial organization that capitalist enterprise creates.” Note that Schumpeter says 
“creates,” not “destroys.” As an example of mutation, Schumpeter cites “the stages of organizational 
development from the craft shop to the factory to a complex corporation like U.S. Steel...” 

Schumpeter understood creative destruction as one unfortunate by-product of a long and complex 
process of creative sustainable change. “Capitalism,” he wrote, “creates and destroys.” Schumpeter was 
adamant on this point: “Creative response shapes the whole course of subsequent events and their ‘long- 
run’ outcome.... Creative response changes social and economic situations for good.... This is why 
creative response is an essential element in the historical process: No deterministic credo avails against 
this.”®° Finally, and contrary to the rhetoric of Silicon Valley and its worship of speed, Schumpeter argued 
that genuine mutation demands patience: “We are dealing with a process whose every element takes 
considerable time in revealing its true features and ultimate effects.... We must judge its performance 
over time, as it unfolds through decades or centuries.” =! 

The significance of a “mutation” in Schumpeter’s reckoning implies a high threshold, one that is 
crossed in time through the serious work of inventing new institutional forms embedded in the new needs 
of new people. Relatively little destruction is creative, especially in the absence of a robust double 
movement. This is illustrated in Schumpeter’s example of US Steel, founded by some of the Gilded Age’s 
most notorious “robber barons,” including Andrew Carnegie and J. P. Morgan. Under pressure from an 
increasingly insistent double movement, US Steel eventually institutionalized fair labor practices through 
unions and collective bargaining as well as internal labor markets, career ladders, professional 
hierarchies, employment security, training, and development, all while implementing its technological 
advances in mass production. 

Mutation is not a fairy tale; it is rational capitalism, bound in reciprocities with its populations through 
democratic institutions. Mutations fundamentally change the nature of capitalism by shifting it in the 
direction of those it is supposed to serve. This sort of thinking is not nearly as sexy or exciting as the 
“boys and their toys” gambit would have us think, but this is what it will take to move the dial of 
economic history beyond the collision and toward modernity. 


VI. Surveillance Capitalism Fills the Void 


A new breed of economic power swiftly filled the void in which every casual search, like, and click was 
claimed as an asset to be tracked, parsed, and monetized by some company, all within a decade of the 
iPod’s debut. It was as if a shark had been silently circling the depths all along, just below the surface of 
the action, only to occasionally leap glistening from the water in pursuit of a fresh bite of flesh. 
Eventually, companies began to explain these violations as the necessary quid pro quo for “free” internet 
services. Privacy, they said, was the price one must pay for the abundant rewards of information, 
connection, and other digital goods when, where, and how you want them. These explanations distracted 
us from the sea change that would rewrite the rules of capitalism and the digital world. 

In retrospect, we can see that the many discordant challenges to users’ expectations were actually tiny 
peepholes into a rapidly emerging institutional form that was learning to exploit second-modernity needs 
and the established norms of “growth through exclusion” as the means to an utterly novel market project. 
Over time, the shark revealed itself as a rapidly multiplying, systemic, internally consistent new variant of 
information capitalism that had set its sights on domination. An unprecedented formulation of capitalism 
was elbowing its way into history: surveillance capitalism. 

This new market form is a unique logic of accumulation in which surveillance is a foundational 
mechanism in the transformation of investment into profit. Its rapid rise, institutional elaboration, and 
significant expansion challenged the tentative promise of the inversion and its advocacy-oriented values. 
More generally, the rise of surveillance capitalism betrayed the hopes and expectations of many 
“netizens” who cherished the emancipatory promise of the networked milieu. 

Surveillance capitalism commandeered the wonders of the digital world to meet our needs for 
effective life, promising the magic of unlimited information and a thousand ways to anticipate our needs 
and ease the complexities of our harried lives. We welcomed it into our hearts and homes with our own 
rituals of hospitality. As we shall explore in detail throughout the coming chapters, thanks to surveillance 
capitalism the resources for effective life that we seek in the digital realm now come encumbered with a 
new breed of menace. Under this new regime, the precise moment at which our needs are met is also the 
precise moment at which our lives are plundered for behavioral data, and all for the sake of others’ gain. 
The result is a perverse amalgam of empowerment inextricably layered with diminishment. In the absence 
of a decisive societal response that constrains or outlaws this logic of accumulation, surveillance 
capitalism appears poised to become the dominant form of capitalism in our time. 

How did this happen? It is a question that we shall return to throughout this book as we accumulate 
new insights and answers. For now we can recognize that over the centuries we have imagined threat in 
the form of state power. This left us wholly unprepared to defend ourselves from new companies with 
imaginative names run by young geniuses that seemed able to provide us with exactly what we yearn for at 
little or no cost. This new regime’s most poignant harms, now and later, have been difficult to grasp or 
theorize, blurred by extreme velocity and camouflaged by expensive and illegible machine operations, 
secretive corporate practices, masterful rhetorical misdirection, and purposeful cultural 
misappropriation. On this road, terms whose meanings we take to be positive or at least banal—“the open 
internet,” “interoperability,” and “connectivity’—have been quietly harnessed to a market process in 
which individuals are definitively cast as the means to others’ market ends. 

Surveillance capitalism has taken root so quickly that, with the exception of a courageous cadre of 
legal scholars and technology-savvy activists, it has cunningly managed to evade our understanding and 
agreement. As we will discuss in more depth in Chapter 4, surveillance capitalism is inconceivable 
outside the digital milieu, but neoliberal ideology and policy also provided the habitat in which 


surveillance capitalism could flourish. This ideology and its practical implementation bends second- 
modernity individuals to the draconian quid pro quo at the heart of surveillance capitalism’s logic of 
accumulation, in which information and connection are ransomed for the lucrative behavioral data that 
fund its immense growth and profits. Any effort to interrupt or dismantle surveillance capitalism will have 
to contend with this larger institutional landscape that protects and sustains its operations. 

History offers no control groups, and we cannot say whether with different leadership, more time, or 
other altered circumstances Apple might have perceived, elaborated, and institutionalized the jewel in its 
crown as Henry Ford and Alfred Sloan had done in another era. Nor is that opportunity forever lost—far 
from it. We may yet see the founding of a new synthesis for a third modernity in which a genuine inversion 
and its social compact are institutionalized as principles of a new rational digital capitalism aligned with 
a society of individuals and supported by democratic institutions. The fact that Schumpeter reckoned the 
time line for such institutionalization in decades or even centuries lingers as a critical commentary on our 
larger story. 

These developments are all the more dangerous because they cannot be reduced to known harms— 
monopoly, privacy—and therefore do not easily yield to known forms of combat. The new harms we face 
entail challenges to the sanctity of the individual, and chief among these challenges I count the elemental 
rights that bear on individual sovereignty, including the right to the future tense and the right to 
sanctuary. Each of these rights invokes claims to individual agency and personal autonomy as essential 
prerequisites to freedom of will and to the very concept of democratic order. 

Right now, however, the extreme asymmetries of knowledge and power that have accrued to 
surveillance capitalism abrogate these elemental rights as our lives are unilaterally rendered as data, 
expropriated, and repurposed in new forms of social control, all of it in the service of others’ interests 
and in the absence of our awareness or means of combat. We have yet to invent the politics and new forms 
of collaborative action—this century’s equivalent of the social movements of the late nineteenth and 
twentieth centuries that aimed to tether raw capitalism to society—that effectively assert the people’s 
right to a human future. And while the work of these inventions awaits us, this mobilization and the 
resistance it engenders will define a key battleground upon which the fight for a human future unfolds. 

On August 9, 2011, events ricocheted between two wildly different visions of a third modernity. One 
was based on the digital promise of democratized information in the context of individualized economic 
and social relations. The other reflected the harsh truths of mass exclusion and elite rule. But the lessons 
of that day had not yet been fully tallied when fresh answers—or, more modestly, the tenuous glimmers of 
answers as fragile as a newborn’s translucent skin—rose to the surface of the world’s attention gliding on 
scented ribbons of Spanish lavender and vanilla. 


VU. For a Human Future 


In the wee hours of August 9, 2011, eighteen-year-old Maria Elena Montes sat on the cool marble floor of 
her family’s century-old pastry shop in the El Raval section of Barcelona, nursing her cup of sweet café 
con leche, lulled by the sunrise scuffling of the pigeons in the plaza as she waited for her trays of rum- 
soaked gypsy cakes to set. 

Pasteleria La Dulce occupied a cramped medieval building tucked into a tiny square on one of the few 
streets that had escaped both the wrecking ball and the influx of yuppie chic. The Montes family took care 
that the passing decades had no visible effect on their cherished bakery. Each morning they lovingly filled 
sparkling glass cases with crispy sugar-studded churros, delicate bufuelos fat with vanilla custard, tiny 


paper ramekins of strawberry flan, buttery mantecados, coiled ensaimadas drenched in powdered sugar, 
fluffy magdalenas, crunchy pestifos, and Great-Grandmother Montes’s special fla6, a cake made with 
fresh milk cheese laced with Spanish lavender, fennel, and mint. There were almond and blood-orange 
tarts prepared, according to Sefora Montes, exactly as they had once been served to Queen Isabella. 
Olive-oil ice cream flavored with anise filled the tubs in the gleaming white freezer along the wall. An 
old ceiling fan cycled slowly, nudging the perfume of honey and yeast into every corner of the ageless 
room. 

Only one thing had changed. Any other August would have found Maria Elena and her family at their 
summer cottage nestled into a pine grove near the seaside town of Palafrugell that had been the family’s 
refuge for generations. In 2011, however, neither the Montes nor their customers and friends would take 
their August holidays. The economic crisis had ripped through the country like the black plague, shrinking 
consumption and driving unemployment to 21 percent, the highest in the EU, and to an astonishing 46 
percent among people under twenty-four years old. In Catalonia, the region that includes Barcelona, 18 
percent of its 7.5 million people had fallen below the poverty line.’ In the summer of 2011, few could 
afford the simple pleasure of an August spent by the sea or in the mountains. 

There was new pressure to sell the building and let the future finally swallow La Dulce. The family 
could live comfortably on the proceeds of such a sale, even at the bargain rates they would be forced to 
accept. Business was slow, but Senor Fito Montes refused to lay off any members of a staff that was like 
an extended family after years of steady employment. Just about everyone they knew said that the end was 
inevitable and that the Montes should leap at the opportunity for a dignified exit. But the family was 
determined to make every sacrifice to safeguard Pasteleria La Dulce for the future. 

Just three months earlier, Juan Pablo and Maria had made the pilgrimage to Madrid to join thousands 
of protesters at the Puerta del Sol, where a month-long encampment established Los Indignados, the 15M, 
as the new voice of a people who had finally been pushed to the breaking point by the economics of 
contempt. All that was left to say was “Ya. No mas!” Enough already! The convergence of so many 
citizens in Madrid led to a wave of protests across the nation, and eventually those protests would give 
way to new political parties, including Podemos. Neighborhood assemblies had begun to convene in 
many cities, and the Montes had attended such a meeting in El Raval just the night before. 

With the evening’s conversations still fresh, they gathered in the apartment above the shop in the early 
afternoon of August 9 to share their midday meal and discuss the fate of La Dulce, not quite certain what 
Papa Montes was thinking. 

“The bankers may not know it,” Fito Montes reflected, “but the future will need the past. It will need 
these marble floors and the sweet taste of my gypsy cakes. They treat us like figures in a ledger, like they 
are reading the number of casualties in a plane crash. They believe the future belongs only to them. But 
we each have our story. We each have our life. It is up to us to proclaim our right to the future. The future 
is our home too.” 

Maria and Juan Pablo breathed a shared sigh of relief as they outlined their plan. Juan Pablo would 
withdraw temporarily from his university studies, and Maria Elena would postpone her matriculation. 
They would work on expanding La Dulce’s sales with new home-delivery and catering options. Everyone 
would take a pay cut, but no one would have to leave. Everyone would tighten their belts, except the fat 
bufuelos and their perfect comrades steadfast in neat, delicious rows. 

We know how to challenge the inevitable, they said. We’ve survived wars; we’ve survived the 
Fascists. We’ll survive again. For Fito Montes, his family’s right to anticipate the future as their home 
demanded continuity for some things that are elusive, beautiful, surprising, mysterious, inexpressible, and 
immaterial but without which, they all agreed, life would be mechanical and soulless. He was 


determined, for example, to ensure that another generation of Spanish children would recognize the 
bouquet of his blood-orange tarts flecked with rose petals and thus be awakened to the mystery of 
medieval life in the fragrant gardens of the Alhambra. 

On August 9 the heat rose steadily in the shady square, and the sun emptied the avenues where Huns, 
Moors, Castilians, and Bourbons had each in their turn marched to triumph. Those silent streets bore little 
evidence of the historic deliberations in Madrid that would be featured in the New York Times that very 
day.4 But I imagine the two cities linked by invisible ribbons of scent rising from La Dulce high into the 
bleached Barcelona sky and drifting slowly south and west to settle along the austere facade of the 
building that housed the Agencia Espafiola de Proteccién de Datos, where another struggle for the right to 
the future tense was underway. 

The Spanish Data Protection Agency had chosen to champion the claims of ninety ordinary citizens 
who, like the Montes family, were determined to preserve inherited meaning for a world bent on change at 
the speed of light.8° In the name of “the right to be forgotten,” the Spaniards had stepped into the bullring 
brandishing red capes, resolved to master the fiercest bull of all: Google, the juggernaut of surveillance 
capitalism. When the agency ordered the internet firm to stop indexing the contested links of these ninety 
individuals, the bull received one of its first and most significant blows. 

This official confrontation drew upon the same tenacity, determination, and sentiment that sustained the 
Montes family and millions of other Spaniards compelled to claw back the future from the self- 
proclaimed inevitability of indifferent capital. In the assertion of a right to be forgotten, the complexity of 
human existence, with its thousand million shades of gray, was pitted against surveillance capitalism’s 
economic imperatives that produced the relentless drive to extract and retain information. It was there, in 
Spain, that the right to the future tense was on the move, insisting that the operations of surveillance 
capitalism and its digital architecture are not, never were, and never would be inevitable. Instead, the 
opposition asserted that even Google’s capitalism was made by humans to be unmade and remade by 
democratic processes, not commercial decree. Google’s was not to be the last word on the human or the 
digital future. 

Each of the ninety citizens had a unique claim. One had been terrorized by her former husband and 
didn’t want him to find her address online. Informational privacy was essential to her peace of mind and 
her physical safety. A middle-aged woman was embarrassed by an old arrest from her days as a 
university student. Informational privacy was essential to her identity and sense of dignity. One was an 
attorney, Mario Costeja Gonzalez, who years earlier had suffered the foreclosure of his home. Although 
the matter had long been resolved, a Google search of his name continued to deliver links to the 
foreclosure notice, which, he argued, damaged his reputation. While the Spanish Data Protection Agency 
rejected the idea of requiring newspapers and other originating sites to remove legitimate information— 
such information, they reasoned, would exist somewhere under any circumstances—it endorsed the notion 
that Google had responsibility and should be held to account. After all, Google had unilaterally 
undertaken to change the rules of the information life cycle when it decided to crawl, index, and make 
accessible personal details across the world wide web without asking anyone’s permission. The agency 
concluded that citizens had the right to request the removal of links and ordered Google to stop indexing 
the information and to remove existing links to its original sources. 

Google’s mission to “organize the world’s information and make it universally accessible and 
useful”—starting with the web—changed all of our lives. There have been enormous benefits, to be sure. 
But for individuals it has meant that information that would normally age and be forgotten now remains 
forever young, highlighted in the foreground of each person’s digital identity. The Spanish Data Protection 
Agency recognized that not all information is worthy of immortality. Some information should be forgotten 


because that is only human. Unsurprisingly, Google challenged the agency’s order before the Spanish High 
Court, which selected one of the ninety cases, that of attorney Mario Costeja Gonzalez, for referral to the 
Court of Justice of the European Union. There, after lengthy and dramatic deliberations, the Court of 
Justice announced its decision to assert the right to be forgotten as a fundamental principle of EU law in 
May of 2014.88 

The Court of Justice’s decision, so often reduced to the legal and technical considerations related to 
the deletion or de-linking of personal data, was in fact a key inflection point at which democracy began to 
claw back rights to the future tense from the powerful forces of a new surveillance capitalism determined 
to claim unilateral authority over the digital future. Instead, the court’s analysis claimed the future for the 
human way, rejecting the inevitability of Google’s search-engine technology and recognizing instead that 
search results are the contingent products of the specific economic interests that drive the action from 
within the belly of the machine: “The operator of a search engine is liable to affect significantly the 
fundamental rights to privacy and to the protection of personal data. In the light of the potential 
seriousness of the interference” with those interests, “it cannot be justified by merely the economic 
interest which the operator of such an engine has in that processing.” As legal scholars Paul M. 
Schwartz and Karl-Nikolaus Peifer summarized it, “The Luxembourg Court felt that free flow of 
information matters, but not as much, ultimately, as the safeguarding of dignity, privacy, and data 
protection in the European rights regime.”®° The court conferred upon EU citizens the right to combat, 
requiring Google to establish a process for implementing users’ de-linking requests and authorizing 
citizens to seek recourse in democratic institutions, including “the supervisory authority or the judicial 
authority, so that it carries out the necessary checks and orders the controller to take specific measures 
accordingly.” 22 

In reasserting the right to be forgotten, the court declared that decisive authority over the digital future 
rests with the people, their laws, and their democratic institutions. It affirmed that individuals and 
democratic societies can fight for their rights to the future tense and can win, even in the face of a great 
private power. As the human rights scholar Federico Fabbrini observed, with this vital case the European 
Court of Justice evolved more assertively into the role of a human rights court, stepping into “the mine- 
field of human rights in the digital age....”22 

When the Court of Justice’s decision was announced, the “smart money” said that it could never 
happen in the US, where the internet companies typically seek cover behind the First Amendment as 
justification for their “permissionless innovation.”2! Some technology observers called the ruling 
“nuts.”22 Google’s leaders sneered at the decision. Reporters characterized Google cofounder Sergey 
Brin as “joking” and “dismissive.” When asked about the ruling during a Q&A at a prominent tech 
conference, he said, “I wish we could just forget the ruling.”22 

In response to the ruling, Google CEO and cofounder Larry Page recited the catechism of the firm’s 
mission statement, assuring the Financial Times that the company “still aims to ‘organise the world’s 
information and make it universally accessible and useful.’” Page defended Google’s unprecedented 
information power with an extraordinary statement suggesting that people should trust Google more than 
democratic institutions: “In general, having the data present in companies like Google is better than 
having it in the government with no due process to get that data, because we obviously care about our 
reputation. I’m not sure the government cares about that as much.”24 Speaking to the company’s 
shareholders the day after the court’s ruling, Eric Schmidt characterized the decision as a “balance that 
was struck wrong” in the “collision between a right to be forgotten and a right to know.” 

The comments of Google’s leaders reflected their determination to retain privileged control over the 
future and their indignation at being challenged. However, there was ample evidence that the American 


public did not concede the corporation’s unilateral power. In fact, the smart money appeared not to be all 
that smart. In the year following the EU decision, a national poll of US adults found that 88 percent 
supported a law similar to the right to be forgotten. That year, Pew Research found that 93 percent of 
Americans believed that it was important to have control of “who can get information about you.” A 
series of polls echoed these findings. 

On January 1, 2015, California’s “Online Eraser” law took effect, requiring the operator of a website, 
online service, online application, or mobile application to permit a minor who is a registered user of the 
operator’s service to remove, or to request and obtain removal of, content or information posted by the 
minor. The California law breached a critical surveillance embattlement, attenuating Google’s role as the 
self-proclaimed champion of an unbounded right to know and suggesting that we are still at the beginning, 
not the end, of a long and fitful drama. 

The Spanish Data Protection Agency and later the European Court of Justice demonstrated the 
unbearable lightness of the inevitable, as both institutions declared what is at stake for a human future, 
beginning with the primacy of democratic institutions in shaping a healthy and just digital future. The 
smart money says that US law will never abandon its allegiance to the surveillance capitalists over the 
people. But the next decades may once again prove that the smart money can be wrong. As for the Spanish 
people, their Data Protection Agency, and the European Court of Justice, the passage of time is likely to 
reveal their achievements as a stirring early chapter in the longer story of our fight for a third modern that 
is first and foremost a human future, rooted in an inclusive democracy and committed to the individual’s 
right to effective life. Their message is carefully inscribed for our children to ponder: technological 
inevitability is as light as democracy is heavy, as temporary as the scent of rose petals and the taste of 
honey are enduring. 


VIII. Naming and Taming 


Taming surveillance capitalism must begin with careful naming, a symbiosis that was vividly illustrated 
in the recent history of HIV research, and I offer it as an analogy. For three decades, scientists aimed to 
create a vaccine that followed the logic of earlier cures, training the immune system to produce 
neutralizing antibodies, but mounting data revealed unanticipated behaviors of the HIV virus that defy the 
patterns of other infectious diseases. 

The tide began to turn at the International AIDS Conference in 2012, when new strategies were 
presented that rely on a close understanding of the biology of rare HIV carriers whose blood produces 
natural antibodies. Research began to shift toward methods that reproduce this self-vaccinating 
response.’ As a leading researcher announced, “We know the face of the enemy now, and so we have 
some real clues about how to approach the problem.”22 

The point for us is that every successful vaccine begins with a close understanding of the enemy 
disease. The mental models, vocabularies, and tools distilled from past catastrophes obstruct progress. 
We smell smoke and rush to close doors to rooms that are already fated to vanish. The result is like 
hurling snowballs at a smooth marble wall only to watch them slide down its facade, leaving nothing but a 
wet smear: a fine paid here, an operational detour there, a new encryption package there. 

What is crucial now is that we identify this new form of capitalism on its own terms and in its own 
words. This pursuit necessarily returns us to Silicon Valley, where things move so fast that few people 
know what just happened. It is the habitat for progress “at the speed of dreams,” as one Google engineer 
vividly describes it22 My aim here is to slow down the action in order to enlarge the space for such 


debate and unmask the tendencies of these new creations as they amplify inequality, intensify social 
hierarchy, exacerbate exclusion, usurp rights, and strip personal life of whatever it is that makes it 
personal for you or for me. If the digital future is to be our home, then it is we who must make it so. We 
will need to know. We will need to decide. We will need to decide who decides. This is our fight for a 
human future. 


CHAPTER THREE 


THE DISCOVERY OF BEHAVIORAL SURPLUS 


He watched the stars and noted birds in flight; 
Ariver flooded or a fortress fell: 
He made predictions that were sometimes right; 
His lucky guesses were rewarded well. 


—W. H. AUDEN 
SONNETS FROM CHINA, VI 


I. Google: The Pioneer of Surveillance Capitalism 


Google is to surveillance capitalism what the Ford Motor Company and General Motors were to mass- 
production—based managerial capitalism. New economic logics and their commercial models are 
discovered by people in a time and place and then perfected through trial and error. In our time Google 
became the pioneer, discoverer, elaborator, experimenter, lead practitioner, role model, and diffusion hub 
of surveillance capitalism. GM and Ford’s iconic status as pioneers of twentieth-century capitalism made 
them enduring objects of scholarly research and public fascination because the lessons they had to teach 
resonated far beyond the individual companies. Google’s practices deserve the same kind of examination, 
not merely as a critique of a single company but rather as the starting point for the codification of a 
powerful new form of capitalism. 

With the triumph of mass production at Ford and for decades thereafter, hundreds of researchers, 
businesspeople, engineers, journalists, and scholars would excavate the circumstances of its invention, 
origins, and consequences.! Decades later, scholars continued to write extensively about Ford, the man 
and the company. GM has also been an object of intense scrutiny. It was the site of Peter Drucker’s field 
studies for his seminal Concept of the Corporation, the 1946 book that codified the practices of the 
twentieth-century business organization and established Drucker’s reputation as a management sage. In 
addition to the many works of scholarship and analysis on these two firms, their own leaders 
enthusiastically articulated their discoveries and practices. Henry Ford and his general manager, James 
Couzens, and Alfred Sloan and his marketing man, Henry “Buck” Weaver, reflected on, conceptualized, 
and proselytized their achievements, specifically locating them in the evolutionary drama of American 
capitalism.2 

Google is a notoriously secretive company, and one is hard-pressed to imagine a Drucker equivalent 


freely roaming the scene and scribbling in the hallways. Its executives carefully craft their messages of 
digital evangelism in books and blog posts, but its operations are not easily accessible to outside 
researchers or journalists.4 In 2016 a lawsuit brought against the company by a product manager alleged 
an internal spying program in which employees are expected to identify coworkers who violate the firm’s 
confidentiality agreement: a broad prohibition against divulging anything about the company to anyone.2 
The closest thing we have to a Buck Weaver or James Couzens codifying Google’s practices and 
objectives is the company’s longtime chief economist, Hal Varian, who aids the cause of understanding 
with scholarly articles that explore important themes. Varian has been described as “the Adam Smith of 
the discipline of Googlenomics” and the “godfather” of its advertising model.® It is in Varian’s work that 
we find hidden-in-plain-sight important clues to the logic of surveillance capitalism and its claims to 
power. 

In two extraordinary articles in scholarly journals, Varian explored the theme of “computer-mediated 
transactions” and their transformational effects on the modern economy.’ Both pieces are written in 
amiable, down-to-earth prose, but Varian’s casual understatement stands in counterpoint to his often- 
startling declarations: “Nowadays there is a computer in the middle of virtually every transaction... now 
that they are available these computers have several other uses.”® He then identifies four such new uses: 
“data extraction and analysis,” “new contractual forms due to better monitoring,” “personalization and 
customization,” and “continuous experiments.” 

Varian’s discussions of these new “uses” are an unexpected guide to the strange logic of surveillance 
capitalism, the division of learning that it shapes, and the character of the information civilization toward 
which it leads. We will return to Varian’s observations from time to time in the course of our examination 
of the foundations of surveillance capitalism, aided by a kind of “reverse engineering” of his assertions, 
so that we might grasp the worldview and methods of surveillance capitalism through this lens. “Data 
extraction and analysis,” Varian writes, “is what everyone is talking about when they talk about big data.” 
“Data” are the raw material necessary for surveillance capitalism’s novel manufacturing processes. 
“Extraction” describes the social relations and material infrastructure with which the firm asserts 
authority over those raw materials to achieve economies of scale in its raw-material supply operations. 
“Analysis” refers to the complex of highly specialized computational systems that I will generally refer to 
in these chapters as “machine intelligence.” I like this umbrella phrase because it trains us on the forest 
rather than the trees, helping us decenter from technology to its objectives. But in choosing this phrase I 
also follow Google’s lead. The company describes itself “at the forefront of innovation in machine 
intelligence,” a term in which it includes machine learning as well as “classical” algorithmic production, 
along with many computational operations that are often referred to with other terms such as “predictive 
analytics” or “artificial intelligence.” Among these operations Google cites its work on language 
translation, speech recognition, visual processing, ranking, statistical modeling, and prediction: “In all of 
those tasks and many others, we gather large volumes of direct or indirect evidence of relationships of 
interest, applying learning algorithms to understand and generalize.”?2 These machine intelligence 
operations convert raw material into the firm’s highly profitable algorithmic products designed to predict 
the behavior of its users. The inscrutability and exclusivity of these techniques and operations are the 
moat that surrounds the castle and secures the action within. 

Google’s invention of targeted advertising paved the way to financial success, but it also laid the 
cornerstone of a more far-reaching development: the discovery and elaboration of surveillance 
capitalism. Its business is characterized as an advertising model, and much has been written about 
Google’s automated auction methods and other aspects of its inventions in the field of online advertising. 
With so much verbiage, these developments are both over-described and under-theorized. Our aim in this 
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chapter and those that follow in Part I is to reveal the “laws of motion” that drive surveillance 
competition, and in order to do this we begin by looking freshly at the point of origin, when the 
foundational mechanisms of surveillance capitalism were first discovered. 

Before we begin, I want to say a word about vocabulary. Any confrontation with the unprecedented 
requires new language, and I introduce new terms when existing language fails to capture a new 
phenomenon. Sometimes, however, I intentionally repurpose familiar language because I want to stress 
certain continuities in the function of an element or process. This is the case with “laws of motion,” 
borrowed from Newton’s laws of inertia, force, and equal and opposite reactions. 

Over the years historians have adopted this term to describe the “laws” of industrial capitalism. For 
example, economic historian Ellen Meiksins Wood documents the origins of capitalism in the changing 
relations between English property owners and tenant farmers, as the owners began to favor productivity 
over coercion: “The new historical dynamic allows us to speak of ‘agrarian capitalism’ in early modern 
England, a social form with distinctive ‘laws of motion’ that would eventually give rise to capitalism in 
its mature, industrial form.”!2 Wood describes how the new “laws of motion” eventually manifested 
themselves in industrial production: 


The critical factor in the divergence of capitalism from all other forms of “commercial society” was the development of certain social 
property relations that generated market imperatives and capitalist “laws of motion”... competitive production and _profit- 
maximization, the compulsion to reinvest surpluses, and the relentless need to improve labour-productivity associated with capitalism. 


... Those laws of motion required vast social transformations and upheavals to set them in train. They required a transformation in 


the human metabolism with nature, in the provision of life’s basic ne cessities 14 


My argument here is that although surveillance capitalism does not abandon established capitalist 
“laws” such as competitive production, profit maximization, productivity, and growth, these earlier 
dynamics now operate in the context of a new logic of accumulation that also introduces its own 
distinctive laws of motion. Here and in following chapters, we will examine these foundational dynamics, 
including surveillance capitalism’s idiosyncratic economic imperatives defined by extraction and 
prediction, its unique approach to economies of scale and scope in raw-material supply, its necessary 
construction and elaboration of means of behavioral modification that incorporate its machine- 
intelligence—based “means of production” in a more complex system of action, and the ways in which the 
requirements of behavioral modification orient all operations toward totalities of information and control, 
creating the framework for an unprecedented instrumentarian power and its societal implications. For 
now, my aim is to reconstruct our appreciation of familiar ground through new lenses: Google’s early 
days of optimism, crisis, and invention. 


IT. A Balance of Power 


Google was incorporated in 1998, founded by Stanford graduate students Larry Page and Sergey Brin just 
two years after the Mosaic browser threw open the doors of the world wide web to the computer-using 
public. From the start, the company embodied the promise of information capitalism as a liberating and 
democratic social force that galvanized and delighted second-modernity populations around the world. 
Thanks to this wide embrace, Google successfully imposed computer mediation on broad new 
domains of human behavior as people searched online and engaged with the web through a growing roster 
of Google services. As these new activities were informated for the first time, they produced wholly new 


data resources. For example, in addition to key words, each Google search query produces a wake of 
collateral data such as the number and pattern of search terms, how a query is phrased, spelling, 
punctuation, dwell times, click patterns, and location. 

Early on, these behavioral by-products were haphazardly stored and operationally ignored. Amit 
Patel, a young Stanford graduate student with a special interest in “data mining,” is frequently credited 
with the groundbreaking insight into the significance of Google’s accidental data caches. His work with 
these data logs persuaded him that detailed stories about each user—thoughts, feelings, interests—could 
be constructed from the wake of unstructured signals that trailed every online action. These data, he 
concluded, actually provided a “broad sensor of human behavior” and could be put to immediate use in 
realizing cofounder Larry Page’s dream of Search as a comprehensive artificial intelligence.“ 

Google’s engineers soon grasped that the continuous flows of collateral behavioral data could turn the 
search engine into a recursive learning system that constantly improved search results and spurred product 
innovations such as spell check, translation, and voice recognition. As Kenneth Cukier observed at that 
time, 


Other search engines in the 1990s had the chance to do the same, but did not pursue it. Around 2000 Yahoo! saw the potential, but 
nothing came of the idea. It was Google that recognized the gold dust in the detritus of its interactions with its users and took the 


trouble to collect it up.... Google exploits information that is a by-product of user interactions, or data exhaust, which is automatically 


recycled to improve the service or create an entirely new product.42 


What had been regarded as waste material—‘“data exhaust” spewed into Google’s servers during the 
combustive action of Search—was quickly reimagined as a critical element in the transformation of 
Google’s search engine into a reflexive process of continuous learning and improvement. 

At that early stage of Google’s development, the feedback loops involved in improving its Search 
functions produced a balance of power: Search needed people to learn from, and people needed Search to 
learn from. This symbiosis enabled Google’s algorithms to learn and produce ever-more relevant and 
comprehensive search results. More queries meant more learning; more learning produced more 
relevance. More relevance meant more searches and more users./4 By the time the young company held its 
first press conference in 1999, to announce a $25 million equity investment from two of the most revered 
Silicon Valley venture capital firms, Sequoia Capital and Kleiner Perkins, Google Search was already 
fielding seven million requests each day./? A few years later, Hal Varian, who joined Google as its chief 
economist in 2002, would note, “Every action a user performs is considered a signal to be analyzed and 
fed back into the system.”!© The Page Rank algorithm, named after its founder, had already given Google a 
significant advantage in identifying the most popular results for queries. Over the course of the next few 
years it would be the capture, storage, analysis, and learning from the by-products of those search queries 
that would turn Google into the gold standard of web search. 

The key point for us rests on a critical distinction. During this early period, behavioral data were put 
to work entirely on the user’s behalf. User data provided value at no cost, and that value was reinvested 
in the user experience in the form of improved services: enhancements that were also offered at no cost to 
users. Users provided the raw material in the form of behavioral data, and those data were harvested to 
improve speed, accuracy, and relevance and to help build ancillary products such as translation. I call this 
the behavioral value reinvestment cycle, in which all behavioral data are reinvested in the improvement 
of the product or service (see Figure 1). 

The cycle emulates the logic of the iPod; it worked beautifully at Google but with one critical 
difference: the absence of a sustainable market transaction. In the case of the iPod, the cycle was triggered 


by the purchase of a high-margin physical product. Subsequent reciprocities improved the iPod product 
and led to increased sales. Customers were the subjects of the commercial process, which promised 
alignment with their “what I want, when I want, where I want” demands. At Google, the cycle was 
similarly oriented toward the individual as its subject, but without a physical product to sell, it floated 
outside the marketplace, an interaction with “users” rather than a market transaction with customers. 

This helps to explain why it is inaccurate to think of Google’s users as its customers: there is no 
economic exchange, no price, and no profit. Nor do users function in the role of workers. When a 
capitalist hires workers and provides them with wages and means of production, the products that they 
produce belong to the capitalist to sell at a profit. Not so here. Users are not paid for their labor, nor do 
they operate the means of production, as we’ll discuss in more depth later in this chapter. Finally, people 
often say that the user is the “product.” This is also misleading, and it is a point that we will revisit more 
than once. For now let’s say that users are not products, but rather we are the sources of raw-material 
supply. As we shall see, surveillance capitalism’s unusual products manage to be derived from our 
behavior while remaining indifferent to our behavior. Its products are about predicting us, without 
actually caring what we do or what is done to us. 

To summarize, at this early stage of Google’s development, whatever Search users inadvertently gave 
up that was of value to the company they also used up in the form of improved services. In this 
reinvestment cycle, serving users with amazing Search results “consumed” all the value that users created 
when they provided extra behavioral data. The fact that users needed Search about as much as Search 
needed users created a balance of power between Google and its populations. People were treated as 
ends in themselves, the subjects of a nonmarket, self-contained cycle that was perfectly aligned with 
Google’s stated mission “to organize the world’s information, making it universally accessible and 
useful.” 
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Figure 1: The Behavioral Value Reinvestment Cycle 


IIT. Search for Capitalism: Impatient Money and the State of Exception 


By 1999, despite the splendor of Google’s new world of searchable web pages, its growing computer 
science capabilities, and its glamorous venture backers, there was no reliable way to turn investors’ 
money into revenue. The behavioral value reinvestment cycle produced a very cool search function, but it 
was not yet capitalism. The balance of power made it financially risky and possibly counterproductive to 
charge users a fee for search services. Selling search results would also have set a dangerous precedent 
for the firm, assigning a price to indexed information that Google’s web crawler had already taken from 
others without payment. Without a device like Apple’s iPod or its digital songs, there were no margins, no 
surplus, nothing left over to sell and turn into revenue. 

Google had relegated advertising to steerage class: its AdWords team consisted of seven people, most 
of whom shared the founders’ general antipathy toward ads. The tone had been set in Sergey Brin and 
Larry Page’s milestone paper that unveiled their search engine conception, “The Anatomy of a Large- 
Scale Hypertextual Web Search Engine,” presented at the 1998 World Wide Web Conference: “We expect 
that advertising funded search engines will be inherently biased towards the advertisers and away from 
the needs of the consumers. This type of bias is very difficult to detect but could still have a significant 
effect on the market... we believe the issue of advertising causes enough mixed incentives that it is 
crucial to have a competitive search engine that is transparent and in the academic realm.” “ 

Google’s first revenues depended instead on exclusive licensing deals to provide web services to 
portals such as Yahoo! and Japan’s BIGLOBE.“ It also generated modest revenue from sponsored ads 
linked to search query keywords./2 There were other models for consideration. Rival search engines such 
as Overture, used exclusively by the then-giant portal AOL, or Inktomi, the search engine adopted by 
Microsoft, collected revenues from the sites whose pages they indexed. Overture was also successful in 
attracting online ads with its policy of allowing advertisers to pay for high-ranking search listings, the 
very format that Brin and Page scorned.72 

Prominent analysts publicly doubted whether Google could compete with its more-established rivals. 
As the New York Times asked, “Can Google create a business model even remotely as good as its 
technology?”~! A well-known Forrester Research analyst proclaimed that there were only a few ways for 
Google to make money with Search: “build a portal [like Yahoo!]... partner with a portal... license the 
technology... wait for a big company to purchase them.”22 

Despite these general misgivings about Google’s viability, the firm’s prestigious venture backing gave 
the founders confidence in their ability to raise money. This changed abruptly in April 2000, when the 
legendary dot-com economy began its steep plunge into recession, and Silicon Valley’s Garden of Eden 
unexpectedly became the epicenter of a financial earthquake. 

By mid-April, Silicon Valley’s fast-money culture of privilege was under siege with the implosion of 
what came to be known as the “dot-com bubble.” It is easy to forget exactly how terrifying things were for 
the valley’s ambitious young people and their slightly older investors. Startups with outsized valuations 
just months earlier were suddenly forced to shutter. Prominent articles such as “Doom Stalks the 
Dotcoms” noted that the stock prices of Wall Street’s most-revered internet “high flyers” were “down for 
the count,” with many of them trading below their initial offering price: “With many dotcoms declining, 
neither venture capitalists nor Wall Street is eager to give them a dime....”22 The news brimmed with 
descriptions of shell-shocked investors. The week of April 10 saw the worst decline in the history of the 
NASDAQ, where many internet companies had gone public, and there was a growing consensus that the 
“same” had irreversibly changed.~4 

As the business environment in Silicon Valley unraveled, investors’ prospects for cashing out by 
selling Google to a big company seemed far less likely, and they were not immune to the rising tide of 
panic. Many Google investors began to express doubts about the company’s prospects, and some 


threatened to withdraw support. Pressure for profit mounted sharply, despite the fact that Google Search 
was widely considered the best of all the search engines, traffic to its website was surging, and a 
thousand résumés flooded the firm’s Mountain View office each day. Page and Brin were seen to be 
moving too slowly, and their top venture capitalists, John Doerr from Kleiner Perkins and Michael Moritz 
from Sequoia, were frustrated. According to Google chronicler Steven Levy, 

“The VCs were screaming bloody murder. Tech’s salad days were over, and it wasn’t certain that 
Google would avoid becoming another crushed radish.” 

The specific character of Silicon Valley’s venture funding, especially during the years leading up to 
dangerous levels of startup inflation, also contributed to a growing sense of emergency at Google. As 
Stanford sociologist Mark Granovetter and his colleague Michel Ferrary found in their study of valley 
venture firms, “A connection with a high-status VC firm signals the high status of the startup and 
encourages other agents to link to it.”22 These themes may seem obvious now, but it is useful to mark the 
anxiety of those months of sudden crisis. Prestigious risk investment functioned as a form of vetting— 
much like acceptance to a top university sorts and legitimates students, elevating a few against the 
backdrop of the many—especially in the “uncertain” environment characteristic of high-tech investing. 
Loss of that high-status signaling power assigned a young company to a long list of also-rans in Silicon 
Valley’s fast-moving saga. 

Other research findings point to the consequences of the impatient money that flooded the valley as 
inflationary hype drew speculators and ratcheted up the volatility of venture funding.*® Studies of pre- 
bubble investment patterns showed a “big-score” mentality in which bad results tended to stimulate 
increased investing as funders chased the belief that some young company would suddenly discover the 
elusive business model destined to turn all their bets into rivers of gold.2? Startup mortality rates in 
Silicon Valley outstripped those for other venture capital centers such as Boston and Washington, DC, 
with impatient money producing a few big wins and many losses.“2 Impatient money is also reflected in 
the size of Silicon Valley startups, which during this period were significantly smaller than in other 
regions, employing an average of 68 employees as compared to an average of 112 in the rest of the 
country.2! This reflects an interest in quick returns without spending much time on growing a business or 
deepening its talent base, let alone developing the institutional capabilities that Joseph Schumpeter would 
have advised. These propensities were exacerbated by the larger Silicon Valley culture, where net worth 
was celebrated as the sole measure of success for valley parents and their children.“4 

For all their genius and principled insights, Brin and Page could not ignore the mounting sense of 
emergency. By December 2000, the Wall Street Journal reported on the new “mantra” emerging from 
Silicon Valley’s investment community: “Simply displaying the ability to make money will not be enough 
to remain a major player in the years ahead. What will be required will be an ability to show sustained 
and exponential profits.” 


IV. The Discovery of Behavioral Surplus 


The declaration of a state of exception functions in politics as cover for the suspension of the rule of law 
and the introduction of new executive powers justified by crisis.24 At Google in late 2000, it became a 
rationale for annulling the reciprocal relationship that existed between Google and its users, steeling the 
founders to abandon their passionate and public opposition to advertising. As a specific response to 
investors’ anxiety, the founders tasked the tiny AdWords team with the objective of looking for ways to 
make more money.2 Page demanded that the whole process be simplified for advertisers. In this new 


approach, he insisted that advertisers “shouldn’t even get involved with choosing keywords—Google 
would choose them.” 

Operationally, this meant that Google would turn its own growing cache of behavioral data and its 
computational power and expertise toward the single task of matching ads with queries. New rhetoric 
took hold to legitimate this unusual move. If there was to be advertising, then it had to be “relevant” to 
users. Ads would no longer be linked to keywords in a search query, but rather a particular ad would be 
“targeted” to a particular individual. Securing this holy grail of advertising would ensure relevance to 
users and value to advertisers. 

Absent from the new rhetoric was the fact that in pursuit of this new aim, Google would cross into 
virgin territory by exploiting sensitivities that only its exclusive and detailed collateral behavioral data 
about millions and later billions of users could reveal. To meet the new objective, the behavioral value 
reinvestment cycle was rapidly and secretly subordinated to a larger and more complex undertaking. The 
raw materials that had been solely used to improve the quality of search results would now also be put to 
use in the service of targeting advertising to individual users. Some data would continue to be applied to 
service improvement, but the growing stores of collateral signals would be repurposed to improve the 
profitability of ads for both Google and its advertisers. These behavioral data available for uses beyond 
service improvement constituted a surplus, and it was on the strength of this behavioral surplus that the 
young company would find its way to the “sustained and exponential profits” that would be necessary for 
survival. Thanks to a perceived emergency, a new mutation began to gather form and quietly slip its 
moorings in the implicit advocacy-oriented social contract of the firm’s original relationship with users. 

Google’s declared state of exception was the backdrop for 2002, the watershed year during which 
surveillance capitalism took root. The firm’s appreciation of behavioral surplus crossed another threshold 
that April, when the data logs team arrived at their offices one morning to find that a peculiar phrase had 
surged to the top of the search queries: “Carol Brady’s maiden name.” Why the sudden interest in a 1970s 
television character? It was data scientist and logs team member Amit Patel who recounted the event to 
the New York Times, noting, “You can’t interpret it unless you know what else is going on in the world.” 

The team went to work to solve the puzzle. First, they discerned that the pattern of queries had 
produced five separate spikes, each beginning at forty-eight minutes after the hour. Then they learned that 
the query pattern occurred during the airing of the popular TV show Who Wants to Be a Millionaire? The 
spikes reflected the successive time zones during which the show aired, ending in Hawaii. In each time 
zone, the show’s host posed the question of Carol Brady’s maiden name, and in each zone the queries 
immediately flooded into Google’s servers. 

As the New York Times reported, “The precision of the Carol Brady data was eye-opening for some.” 
Even Brin was stunned by the clarity of Search’s predictive power, revealing events and trends before 
they “hit the radar” of traditional media. As he told the Times, “It was like trying an electron microscope 
for the first time. It was like a moment-by-moment barometer.”=°= Google executives were described by 
the Times as reluctant to share their thoughts about how their massive stores of query data might be 
commercialized. “There is tremendous opportunity with this data,” one executive confided.2 

Just a month before the Carol Brady moment, while the AdWords team was already working on new 
approaches, Brin and Page hired Eric Schmidt, an experienced executive, engineer, and computer science 
Ph.D., as chairman. By August, they appointed him to the CEO’s role. Doerr and Moritz had been pushing 
the founders to hire a professional manager who would know how to pivot the firm toward profit.@ 
Schmidt immediately implemented a “belt-tightening” program, grabbing the budgetary reins and 
heightening the general sense of financial alarm as fund-raising prospects came under threat. A squeeze on 
workspace found him unexpectedly sharing his office with none other than Amit Patel. 


Schmidt later boasted that as a result of their close quarters over the course of several months, he had 
instant access to better revenue figures than did his own financial planners.“! We do not know (and may 
never know) what other insights Schmidt might have gleaned from Patel about the predictive power of 
Google’s behavioral data stores, but there is no doubt that a deeper grasp of the predictive power of data 
quickly shaped Google’s specific response to financial emergency, triggering the crucial mutation that 
ultimately turned AdWords, Google, the internet, and the very nature of information capitalism toward an 
astonishingly lucrative surveillance project. 

Google’s earliest ads had been considered more effective than most online advertising at the time 
because they were linked to search queries and Google could track when users actually clicked on an ad, 
known as the “click-through” rate. Despite this, advertisers were billed in the conventional manner 
according to how many people viewed an ad. As Search expanded, Google created the self-service 
system called AdWords, in which a search that used the advertiser’s keyword would include that 
advertiser’s text box and a link to its landing page. Ad pricing depended upon the ad’s position on the 
search results page. 

Rival search startup Overture had developed an online auction system for web page placement that 
allowed it to scale online advertising targeted to keywords. Google would produce a transformational 
enhancement to that model, one that was destined to alter the course of information capitalism. As a 
Bloomberg journalist explained in 2006, “Google maximizes the revenue it gets from that precious real 
estate by giving its best position to the advertiser who is likely to pay Google the most in total, based on 
the price per click multiplied by Google’s estimate of the likelihood that someone will actually click on 
the ad.”*2 That pivotal multiplier was the result of Google’s advanced computational capabilities trained 
on its most significant and secret discovery: behavioral surplus. From this point forward, the combination 
of ever-increasing machine intelligence and ever-more-vast supplies of behavioral surplus would become 
the foundation of an unprecedented logic of accumulation. Google’s reinvestment priorities would shift 
from merely improving its user offerings to inventing and institutionalizing the most far-reaching and 
technologically advanced raw-material supply operations that the world had ever seen. Henceforth, 
revenues and growth would depend upon more behavioral surplus. 

Google’s many patents filed during those early years illustrate the explosion of discovery, 
inventiveness, and complexity detonated by the state of exception that led to these crucial innovations and 
the firm’s determination to advance the capture of behavioral surplus.*? Among these efforts, I focus here 
on one patent submitted in 2003 by three of the firm’s top computer scientists and titled “Generating User 
Information for Use in Targeted Advertising.”“ The patent is emblematic of the new mutation and the 
emerging logic of accumulation that would define Google’s success. Of even greater interest, it also 
provides an unusual glimpse into the “economic orientation” baked deep into the technology cake by 
reflecting the mindset of Google’s distinguished scientists as they harnessed their knowledge to the firm’s 
new aims.“ In this way, the patent stands as a treatise on a new political economics of clicks and its 
moral universe, before the company learned to disguise this project in a fog of euphemism. 

The patent reveals a pivoting of the backstage operation toward Google’s new audience of genuine 
customers. “The present invention concerns advertising,” the inventors announce. Despite the enormous 
quantity of demographic data available to advertisers, the scientists note that much of an ad budget “is 
simply wasted... it is very difficult to identify and eliminate such waste.”“° 

Advertising had always been a guessing game: art, relationships, conventional wisdom, standard 
practice, but never “science.” The idea of being able to deliver a particular message to a particular 
person at just the moment when it might have a high probability of actually influencing his or her behavior 
was, and had always been, the holy grail of advertising. The inventors point out that online ad systems had 


also failed to achieve this elusive goal. The then-predominant approaches used by Google’s competitors, 
in which ads were targeted to keywords or content, were unable to identify relevant ads “for a particular 
user.” Now the inventors offered a scientific solution that exceeded the most-ambitious dreams of any 
advertising executive: 


There is a need to increase the relevancy of ads served for some user request, such as a search query or a document request... to 


the user that submitted the request.... The present invention may involve novel methods, apparatus, message formats and/or data 


structures for determining user profile information and using such determined user profile information for ad serving. 42 


In other words, Google would no longer mine behavioral data strictly to improve service for users but 
rather to read users’ minds for the purposes of matching ads to their interests, as those interests are 
deduced from the collateral traces of online behavior. With Google’s unique access to behavioral data, it 
would now be possible to know what a particular individual in a particular time and place was thinking, 
feeling, and doing. That this no longer seems astonishing to us, or perhaps even worthy of note, is 
evidence of the profound psychic numbing that has inured us to a bold and unprecedented shift in 
capitalist methods. 

The techniques described in the patent meant that each time a user queries Google’s search engine, the 
system simultaneously presents a specific configuration of a particular ad, all in the fraction of a moment 
that it takes to fulfill the search query. The data used to perform this instant translation from query to ad, a 
predictive analysis that was dubbed “matching,” went far beyond the mere denotation of search terms. 
New data sets were compiled that would dramatically enhance the accuracy of these predictions. These 
data sets were referred to as “user profile information” or “UPI.” These new data meant that there would 
be no more guesswork and far less waste in the advertising budget. Mathematical certainty would replace 
all of that. 

Where would UPI come from? The scientists announce a breakthrough. They first explain that some of 
the new data can be culled from the firm’s existing systems with its continuously accruing caches of 
behavioral data from Search. Then they stress that even more behavioral data can be hunted and herded 
from anywhere in the online world. UPI, they write, “may be inferred,” “presumed,” and “deduced.” 
Their new methods and computational tools could create UPI from integrating and analyzing a user’s 
search patterns, document inquiries, and myriad other signals of online behaviors, even when users do not 
directly provide that personal information: “User profile information may include any information about 
an individual user or a group of users. Such information may be provided by the user, provided by a third- 
party authorized to release user information, and/or derived from user actions. Certain user information 
can be deduced or presumed using other user information of the same user and/or user information of 
other users. UPI may be associated with various entities.” 

The inventors explain that UPI can be deduced directly from a user’s or group’s actions, from any kind 
of document a user views, or from an ad landing page: “For example, an ad for prostate cancer screening 
might be limited to user profiles having the attribute ‘male’ and ‘age 45 and over.’”“2 They describe 
different ways to obtain UPI. One relies on “machine learning classifiers” that predict values on a range 
of attributes. “Association graphs” are developed to reveal the relationships among users, documents, 
search queries, and web pages: “user-to-user associations may also be generated.”22 The inventors also 
note that their methods can be understood only among the priesthood of computer scientists drawn to the 
analytic challenges of this new online universe: “The following description is presented to enable one 
skilled in the art to make and use the invention.... Various modifications to the disclosed embodiments 
will be apparent to those skilled in the art....”2! 


Of critical importance to our story is the scientists’ observation that the most challenging sources of 
friction here are social, not technical. Friction arises when users intentionally fail to provide information 
for no other reason than that they choose not to. “Unfortunately, user profile information is not always 
available,” the scientists warn. Users do not always “voluntarily” provide information, or “the user 
profile may be incomplete... and hence not comprehensive, because of privacy considerations, etc.” 

A clear aim of the patent is to assure its audience that Google scientists will not be deterred by users’ 
exercise of decision rights over their personal information, despite the fact that such rights were an 
inherent feature of the original social contract between the company and its users.22 Even when users do 
provide UPI, the inventors caution, “it may be intentionally or unintentionally inaccurate, it may become 
stale.... UPI for a user... can be determined (or updated or extended) even when no explicit information 
is given to the system.... An initial UPI may include some expressly entered UPI information, though it 
doesn’t need to.” 

The scientists thus make clear that they are willing—and that their inventions are able—to overcome 
the friction entailed in users’ decision rights. Google’s proprietary methods enable it to surveil, capture, 
expand, construct, and claim behavioral surplus, including data that users intentionally choose not to 
share. Recalcitrant users will not be obstacles to data expropriation. No moral, legal, or social 
constraints will stand in the way of finding, claiming, and analyzing others’ behavior for commercial 
purposes. 

The inventors provide examples of the kinds of attributes that Google could assess as it compiles its 
UPI data sets while circumnavigating users’ knowledge, intentions, and consent. These include websites 
visited, psychographics, browsing activity, and information about previous advertisements that the user 
has been shown, selected, and/or made purchases after viewing. It is a long list that is certainly much 
longer today. 

Finally, the inventors observe another obstacle to effective targeting. Even when user information 
exists, they say, “Advertisers may not be able to use this information to target ads effectively.”2° On the 
strength of the invention presented in this patent, and others related to it, the inventors publicly declare 
Google’s unique prowess in hunting, capturing, and transforming surplus into predictions for accurate 
targeting. No other firm could equal its range of access to behavioral surplus, its bench strength of 
scientific knowledge and technique, its computational power, or its storage infrastructure. In 2003 only 
Google could pull surplus from multiple sites of activity and integrate each increment of data into 
comprehensive “data structures.” Google was uniquely positioned with the state-of-the-art knowledge in 
computer science to convert those data into predictions of who will click on which configuration of what 
ad as the basis for a final “matching” result, all computed in micro-fractions of a second. 

To state all this in plain language, Google’s invention revealed new capabilities to infer and deduce 
the thoughts, feelings, intentions, and interests of individuals and groups with an automated architecture 
that operates as a one-way mirror irrespective of a person’s awareness, knowledge, and consent, thus 
enabling privileged secret access to behavioral data. 

A one-way mirror embodies the specific social relations of surveillance based on asymmetries of 
knowledge and power. The new mode of accumulation invented at Google would derive, above all, from 
the firm’s willingness and ability to impose these social relations on its users. Its willingness was 
mobilized by what the founders came to regard as a state of exception; its ability came from its actual 
success in leveraging privileged access to behavioral surplus in order to predict the behavior of 
individuals now, soon, and later. The predictive insights thus acquired would constitute a world-historic 
competitive advantage in a new marketplace where low-risk bets about the behavior of individuals are 
valued, bought, and sold. 


Google would no longer be a passive recipient of accidental data that it could recycle for the benefit 
of its users. The targeted advertising patent sheds light on the path of discovery that Google traveled from 
its advocacy-oriented founding toward the elaboration of behavioral surveillance as a full-blown logic of 
accumulation. The invention itself exposes the reasoning through which the behavioral value reinvestment 
cycle was subjugated to the service of a new commercial calculation. Behavioral data, whose value had 
previously been “used up” on improving the quality of Search for users, now became the pivotal—and 
exclusive to Google—raw material for the construction of a dynamic online advertising marketplace. 
Google would now secure more behavioral data than it needed to serve its users. That surplus, a 
behavioral surplus, was the game-changing, zero-cost asset that was diverted from service improvement 
toward a genuine and highly lucrative market exchange. 

These capabilities were and remain inscrutable to all but an exclusive data priesthood among whom 
Google is the tibermensch. They operate in obscurity, indifferent to social norms or individual claims to 
self-determining decision rights. These moves established the foundational mechanisms of surveillance 
capitalism. 

The state of exception declared by Google’s founders transformed the youthful Dr. Jekyll into a 
ruthless, muscular Mr. Hyde determined to hunt his prey anywhere, anytime, irrespective of others’ self- 
determining aims. The new Google ignored claims to self-determination and acknowledged no a priori 
limits on what it could find and take. It dismissed the moral and legal content of individual decision rights 
and recast the situation as one of technological opportunism and unilateral power. This new Google 
assures its actual customers that it will do whatever it takes to transform the natural obscurity of human 
desire into scientific fact. This Google is the superpower that establishes its own values and pursues its 
own purposes above and beyond the social contracts to which others are bound. 


V. Surplus at Scale 


There were other new elements that helped to establish the centrality of behavioral surplus in Google’s 
commercial operations, beginning with its pricing innovations. The first new pricing metric was based on 
“click-through rates,” or how many times a user clicks on an ad through to the advertiser’s web page, 
rather than pricing based on the number of views that an ad receives. The click-through was interpreted as 
a signal of relevance and therefore a measure of successful targeting, operational results that derive from 
and reflect the value of behavioral surplus. 

This new pricing discipline established an ever-escalating incentive to increase behavioral surplus in 
order to continuously upgrade the effectiveness of predictions. Better predictions lead directly to more 
click-throughs and thus to revenue. Google learned new ways to conduct automated auctions for ad 
targeting that allowed the new invention to scale quickly, accommodating hundreds of thousands of 
advertisers and billions (later it would be trillions) of auctions simultaneously. Google’s unique auction 
methods and capabilities earned a great deal of attention, which distracted observers from reflecting on 
exactly what was being auctioned: derivatives of behavioral surplus. Click-through metrics 
institutionalized “customer” demand for these prediction products and thus established the central 
importance of economies of scale in surplus supply operations. Surplus capture would have to become 
automatic and ubiquitous if the new logic was to succeed, as measured by the successful trading of 
behavioral futures. 

Another key metric called the “quality score” helped determine the price of an ad and its specific 
position on the page, in addition to advertisers’ own auction bids. The quality score was determined in 


part by click-through rates and in part by the firm’s analyses of behavioral surplus. “The clickthrough rate 
needed to be a predictive thing,” one top executive insisted, and that would require “all the information 
we had about the query right then.” It would take enormous computing power and leading-edge 
algorithmic programs to produce powerful predictions of user behavior that became the criteria for 
estimating the relevance of an ad. Ads that scored high would sell at a lower price than those that scored 
poorly. Google’s customers, its advertisers, complained that the quality score was a black box, and 
Google was determined to keep it so. Nonetheless, when customers followed its disciplines and produced 
high-scoring ads, their click-through rates soared. 

AdWords quickly became so successful that it inspired significant expansion of the surveillance logic. 
Advertisers demanded more clicks.2° The answer was to extend the model beyond Google’s search pages 
and convert the entire internet into a canvas for Google’s targeted ads. This required turning Google’s 
newfound skills at “data extraction and analysis,” as Hal Varian put it, toward the content of any web page 
or user action by employing Google’s rapidly expanding semantic analysis and artificial intelligence 
capabilities to efficiently “squeeze” meaning from them. Only then could Google accurately assess the 
content of a page and how users interact with that content. This “content-targeted advertising” based on 
Google’s patented methods was eventually named AdSense. By 2004, AdSense had achieved a run rate of 
a million dollars per day, and by 2010, it produced annual revenues of more than $10 billion. 

So here was an unprecedented and lucrative brew: behavioral surplus, data science, material 
infrastructure, computational power, algorithmic systems, and automated platforms. This convergence 
produced unprecedented “relevance” and billions of auctions. Click-through rates skyrocketed. Work on 
AdWords and AdSense became just as important as work on Search. 

With click-through rates as the measure of relevance accomplished, behavioral surplus was 
institutionalized as the cornerstone of a new kind of commerce that depended upon online surveillance at 
scale. Insiders referred to Google’s new science of behavioral prediction as the “physics of clicks.”” 
Mastery of this new domain required a specialized breed of click physicists who would secure Google’s 
preeminence within the nascent priesthood of behavioral prediction. The firm’s substantial revenue flows 
summoned the greatest minds of our age from fields such as artificial intelligence, statistics, machine 
learning, data science, and predictive analytics to converge on the prediction of human behavior as 
measured by click-through rates: computer-mediated fortune-telling and selling. The firm would recruit an 
authority on information economics, and consultant to Google since 2001, as the patriarch of this 
auspicious group and the still-young science: Hal Varian was the chosen shepherd of this flock. 

Page and Brin had been reluctant to embrace advertising, but as the evidence mounted that ads could 
save the company from crisis, their attitudes shifted.°° Saving the company also meant saving themselves 
from being just another couple of very smart guys who couldn’t figure out how to make real money, 
insignificant players in the intensely material and competitive culture of Silicon Valley. Page was haunted 
by the example of the brilliant but impoverished scientist Nikola Tesla, who died without ever benefiting 
financially from his inventions. “You need to do more than just invent things,” Page reflected. Brin had 
his own take: “Honestly, when we were still in the dot-com boom days, I felt like a schmuck. I had an 
internet startup—so did everybody else. It was unprofitable, like everybody else’s.”®2 Exceptional threats 
to their financial and social status appear to have awakened a survival instinct in Page and Brin that 
required exceptional adaptive measures.° The Google founders’ response to the fear that stalked their 
community effectively declared a “state of exception” in which it was judged necessary to suspend the 
values and principles that had guided Google’s founding and early practices. 

Later, Sequoia’s Moritz recalled the crisis conditions that provoked the firm’s “ingenious” self- 
reinvention, when crisis opened a fork in the road and drew the company in a wholly new direction. He 


stressed the specificity of Google’s inventions, their origins in emergency, and the 180-degree turn from 
serving users to surveilling them. Most of all, he credited the discovery of behavioral surplus as the 
game-changing asset that turned Google into a fortune-telling giant, pinpointing Google’s breakthrough 
transformation of the Overture model, when the young company first applied its analytics of behavioral 
surplus to predict the likelihood of a click: 


The first 12 months of Google were not a cakewalk, because the company didn’t start off in the business that it eventually tapped. At 
first it went in a different direction, which was selling its technology—selling licenses for its search engines to larger internet 
properties and to corporations.... Cash was going out of the window at a feral rate during the first six, seven months. And then, very 
ingeniously, Larry... and Sergey... and others fastened on a model that they had seen this other company, Overture, develop, which 
was ranked advertisements. They saw how it could be improved and enhanced and made it their own, and that transformed the 


business.&4 


Moritz’s reflections suggest that without the discovery of behavioral surplus and the turn toward 
surveillance operations, Google’s “feral” rate of spending was not sustainable and the firm’s survival 
was imperiled. We will never know what Google might have made of itself without the state of exception 
fueled by the emergency of impatient money that shaped those crucial years of development. What other 
pathways to sustainable revenue might have been explored or invented? What alternative futures might 
have been summoned to keep faith with the founders’ principles and with their users’ rights to self- 
determination? Instead, Google loosed a new incarnation of capitalism upon the world, a Pandora’s box 
whose contents we are only beginning to understand. 


VI. A Human Invention 


Key to our conversation is this fact: surveillance capitalism was invented by a specific group of human 
beings in a specific time and place. It is not an inherent result of digital technology, nor is it a necessary 
expression of information capitalism. It was intentionally constructed at a moment in history, in much the 
Same way that the engineers and tinkerers at the Ford Motor Company invented mass production in the 
Detroit of 1913. 

Henry Ford set out to prove that he could maximize profits by driving up volumes, radically 
decreasing costs, and widening demand. It was an unproven commercial equation for which no economic 
theory or body of practice existed. Fragments of the formula had surfaced before—in meatpacking plants, 
flour-milling operations, sewing machine and bicycle factories, armories, canneries, and breweries. 
There was a growing body of practical knowledge about the interchangeability of parts and absolute 
standardization, precision machines, and continuous flow production. But no one had achieved the grand 
symphony that Ford heard in his imagination. 

As historian David Hounshell tells it, there was a time, April 1, 1913, and a place, Detroit, when the 
first moving assembly line seemed to be “just another step in the years of development at Ford yet 
somehow suddenly dropped out of the sky. Even before the end of the day, some of the engineers sensed 
that they had made a fundamental breakthrough.”© Within a year, productivity increases across the plant 
ranged from 50 percent to as much as ten times the output of the old fixed-assembly methods.© The Model 
T that sold for $825 in 1908 was priced at a record low for a four-cylinder automobile in 1924, just 
$260.92 

Much as with Ford, some elements of the economic surveillance logic in the online environment had 


been operational for years, familiar only to a rarefied group of early computer experts. For example, the 
software mechanism known as the “cookie”—bits of code that allow information to be passed between a 
server and a client computer—was developed in 1994 at Netscape, the first commercial web browser 
company.” Similarly, “web bugs”—tiny (often invisible) graphics embedded in web pages and e-mail 
and designed to monitor user activity and collect personal information—were well-known to experts in 
the late 1990s. 

These experts were deeply concerned about the privacy implications of such monitoring mechanisms, 
and at least in the case of cookies, there were institutional efforts to design internet policies that would 
prohibit their invasive capabilities to monitor and profile users.“ By 1996, the function of cookies had 
become a contested public policy issue. Federal Trade Commission workshops in 1996 and 1997 
discussed proposals that would assign control of all personal information to users by default with a 
simple automated protocol. Advertisers bitterly contested this scheme, collaborating instead to avert 
government regulation by forming a “self-regulating” association known as the Network Advertising 
Initiative. Still, in June 2000 the Clinton administration banned cookies from all federal websites, and by 
April 2001, three bills before Congress included provisions to regulate cookies. 

Google brought new life to these practices. As had occurred at Ford a century earlier, the company’s 
engineers and scientists were the first to conduct the entire commercial surveillance symphony, integrating 
a wide range of mechanisms from cookies to proprietary analytics and algorithmic software capabilities 
in a Sweeping new logic that enshrined surveillance and the unilateral expropriation of behavioral data as 
the basis for a new market form. The impact of this invention was just as dramatic as Ford’s. In 2001, as 
Google’s new systems to exploit its discovery of behavioral surplus were being tested, net revenues 
jumped to $86 million (more than a 400 percent increase over 2000), and the company turned its first 
profit. By 2002, the cash began to flow and has never stopped, definitive evidence that behavioral surplus 
combined with Google’s proprietary analytics were sending arrows to their marks. Revenues leapt to 
$347 million in 2002, then $1.5 billion in 2003, and $3.5 billion in 2004, the year the company went 
public.“ The discovery of behavioral surplus had produced a stunning 3,590 percent increase in revenue 
in less than four years. 


VII. The Secrets of Extraction 


It is important to note the vital differences for capitalism in these two moments of originality at Ford and 
Google. Ford’s inventions revolutionized production. Google’s inventions revolutionized extraction and 
established surveillance capitalism’s first economic imperative: the extraction imperative. The 
extraction imperative meant that raw-material supplies must be procured at an ever-expanding scale. 
Industrial capitalism had demanded economies of scale in production in order to achieve high throughput 
combined with low unit cost. In contrast, surveillance capitalism demands economies of scale in the 
extraction of behavioral surplus. 

Mass production was aimed at new sources of demand in the early twentieth century’s first mass 
consumers. Ford was clear on this point: “Mass production begins in the perception of a public need.” 
Supply and demand were linked effects of the new “conditions of existence” that defined the lives of my 
great-grandparents Sophie and Max and other travelers in the first modernity. Ford’s invention deepened 
the reciprocities between capitalism and these populations. 

In contrast, Google’s inventions destroyed the reciprocities of its original social contract with users. 
The role of the behavioral value reinvestment cycle that had once aligned Google with its users changed 


dramatically. Instead of deepening the unity of supply and demand with its populations, Google chose to 
reinvent its business around the burgeoning demand of advertisers eager to squeeze and scrape online 
behavior by any available means in the competition for market advantage. In the new operation, users 
were no longer ends in themselves but rather became the means to others’ ends. 

Reinvestment in user services became the method for attracting behavioral surplus, and users became 
the unwitting suppliers of raw material for a larger cycle of revenue generation. The scale of surplus 
expropriation that was possible at Google would soon eliminate all serious competitors to its core search 
business as the windfall earnings from leveraging behavioral surplus were used to continuously draw 
more users into its net, thus establishing its de facto monopoly in Search. On the strength of Google’s 
inventions, discoveries, and strategies, it became the mother ship and ideal type of a new economic logic 
based on fortune-telling and selling—an ancient and eternally lucrative craft that has fed on humanity’s 
confrontation with uncertainty from the beginning of the human story. 

It was one thing to proselytize achievements in production, as Henry Ford had done, but quite another 
to boast about the continuous intensification of hidden processes aimed at the extraction of behavioral 
data and personal information. The last thing that Google wanted was to reveal the secrets of how it had 
rewritten its own rules and, in the process, enslaved itself to the extraction imperative. Behavioral 
surplus was necessary for revenue, and secrecy would be necessary for the sustained accumulation of 
behavioral surplus. 

This is how secrecy came to be institutionalized in the policies and practices that govern every aspect 
of Google’s behavior onstage and offstage. Once Google’s leadership understood the commercial power 
of behavioral surplus, Schmidt instituted what he called the “hiding strategy.” Google employees were 
told not to speak about what the patent had referred to as its “novel methods, apparatus, message formats 
and/or data structures” or confirm any rumors about flowing cash. Hiding was not a post hoc strategy; it 
was baked into the cake that would become surveillance capitalism. 

Former Google executive Douglas Edwards writes compellingly about this predicament and the 
culture of secrecy it shaped. According to his account, Page and Brin were “hawks,” insisting on 
aggressive data capture and retention: “Larry opposed any path that would reveal our technological 
secrets or stir the privacy pot and endanger our ability to gather data.” Page wanted to avoid arousing 
users’ curiosity by minimizing their exposure to any clues about the reach of the firm’s data operations. He 
questioned the prudence of the electronic scroll in the reception lobby that displays a continuous stream of 
search queries, and he “tried to kill” the annual Google Zeitgeist conference that summarizes the year’s 
trends in search terms.” 

Journalist John Battelle, who chronicled Google during the 2002-2004 period, described the 
company’s “aloofness,” “limited information sharing,” and “alienating and unnecessary secrecy and 
isolation.” Another early company biographer notes, “What made this information easier to keep is that 
almost none of the experts tracking the business of the internet believed that Google’s secret was even 
possible.” As Schmidt told the New York Times, “You need to win, but you are better off winning 
softly.”22 The scientific and material complexity that supported the capture and analysis of behavioral 
surplus also enabled the hiding strategy, an invisibility cloak over the whole operation. “Managing search 
at our scale is a very serious barrier to entry,” Schmidt warned would-be competitors.“ 

To be sure, there are always sound business reasons for hiding the location of your gold mine. In 
Google’s case, the hiding strategy accrued to its competitive advantage, but there were other reasons for 
concealment and obfuscation. What might the response have been back then if the public were told that 
Google’s magic derived from its exclusive capabilities in unilateral surveillance of online behavior and 
its methods specifically designed to override individual decision rights? Google policies had to enforce 


secrecy in order to protect operations that were designed to be undetectable because they took things from 
users without asking and employed those unilaterally claimed resources to work in the service of others’ 
purposes. 

That Google had the power to choose secrecy is itself testament to the success of its own claims. This 
power is a crucial illustration of the difference between “decision rights” and “privacy.” Decision rights 
confer the power to choose whether to keep something secret or to share it. One can choose the degree of 
privacy or transparency for each situation. US Supreme Court Justice William O. Douglas articulated this 
view of privacy in 1967: “Privacy involves the choice of the individual to disclose or to reveal what he 
believes, what he thinks, what he possesses... 7 

Surveillance capitalism lays claim to these decision rights. The typical complaint is that privacy is 
eroded, but that is misleading. In the larger societal pattern, privacy is not eroded but redistributed, as 
decision rights over privacy are claimed for surveillance capital. Instead of people having the rights to 
decide how and what they will disclose, these rights are concentrated within the domain of surveillance 
capitalism. Google discovered this necessary element of the new logic of accumulation: it must assert the 
rights to take the information upon which its success depends. 

The corporation’s ability to hide this rights grab depends on language as much as it does on technical 
methods or corporate policies of secrecy. George Orwell once observed that euphemisms are used in 
politics, war, and business as instruments that “make lies sound truthful and murder respectable.”®! 
Google has been careful to camouflage the significance of its behavioral surplus operations in industry 
jargon. Two popular terms—“digital exhaust” and “digital breadcrumbs”—connote worthless waste: 
leftovers lying around for the taking.22 Why allow exhaust to drift in the atmosphere when it can be 
recycled into useful data? Who would think to call such recycling an act of exploitation, expropriation, or 
plunder? Who would dare to redefine “digital exhaust” as booty or contraband, or imagine that Google 
had learned how to purposefully construct that so-called “exhaust” with its methods, apparatus, and data 
structures? 

The word “targeted” is another euphemism. It evokes notions of precision, efficiency, and competence. 
Who would guess that targeting conceals a new political equation in which Google’s concentrations of 
computational power brush aside users’ decision rights as easily as King Kong might shoo away an ant, 
all accomplished offstage where no one can see? 

These euphemisms operate in exactly the same way as those found on the earliest maps of the North 
American continent, in which whole regions were labeled with terms such as “heathens,” “infidels,” 
“idolaters,” “primitives,” “vassals,” and “rebels.” On the strength of those euphemisms, native peoples— 
their places and claims—were deleted from the invaders’ moral and legal equations, legitimating the acts 
of taking and breaking that paved the way for church and monarchy. 

The intentional work of hiding naked facts in rhetoric, omission, complexity, exclusivity, scale, 
abusive contracts, design, and euphemism is another factor that helps explain why during Google’s 
breakthrough to profitability, few noticed the foundational mechanisms of its success and their larger 
significance. In this picture, commercial surveillance is not merely an unfortunate accident or occasional 
lapse. It is neither a necessary development of information capitalism nor a necessary product of digital 
technology or the internet. It is a specifically constructed human choice, an unprecedented market form, an 
original solution to emergency, and the underlying mechanism through which a new asset class is created 
on the cheap and converted to revenue. Surveillance is the path to profit that overrides “we the people,” 
taking our decision rights without permission and even when we say “no.” The discovery of behavioral 
surplus marks a critical turning point not only in Google’s biography but also in the history of capitalism. 

In the years following its IPO in 2004, Google’s spectacular financial breakthrough first astonished 


3D 66. 


and then magnetized the online world. Silicon Valley investors had doubled down on risk for years, in 
search of that elusive business model that would make it all worthwhile. When Google’s financial results 
went public, the hunt for mythic treasure was officially over. 

The new logic of accumulation spread first to Facebook, which launched the same year that Google 
went public. CEO Mark Zuckerberg had rejected the strategy of charging users a fee for service as the 
telephone companies had done in an earlier century. “Our mission is to connect every person in the world. 
You don’t do that by having a service people pay for,” he insisted. In May 2007 he introduced the 
Facebook platform, opening up the social network to everyone, not just people with a college e-mail 
address. Six months later, in November, he launched his big advertising product, Beacon, which would 
automatically share transactions from partner websites with all of a user’s “friends.” These posts would 
appear even if the user was not currently logged into Facebook, without the user’s knowledge or an opt-in 
function. The howls of protest—from users but also from some of Facebook’s partners such as Coca-Cola 
—forced Zuckerberg to back down swiftly. By December, Beacon became an opt-in program. The twenty- 
three-year-old CEO understood the potential of surveillance capitalism, but he had not yet mastered 
Google’s facility in obscuring its operations and intent. 

The pressing question in Facebook’s headquarters—“How do we turn all those Facebook users into 
money?”—still required an answer.®2 In March 2008, just three months after having to kill his first attempt 
at emulating Google’s logic of accumulation, Zuckerberg hired Google executive Sheryl Sandberg to be 
Facebook’s chief operating officer. The onetime chief of staff to US Treasury Secretary Larry Summers, 
Sandberg had joined Google in 2001, ultimately rising to be its vice president of global online sales and 
operations. At Google she led the development of surveillance capitalism through the expansion of 
AdWords and other aspects of online sales operations.2° One investor who had observed the company’s 
growth during that period concluded, “Sheryl created AdWords.”®2 

In signing on with Facebook, the talented Sandberg became the “Typhoid Mary” of surveillance 
capitalism as she led Facebook’s transformation from a social networking site to an advertising behemoth. 
Sandberg understood that Facebook’s social graph represented an awe-inspiring source of behavioral 
surplus: the extractor’s equivalent of a nineteenth-century prospector stumbling into a valley that sheltered 
the largest diamond mine and the deepest gold mine ever to be discovered. “We have better information 
than anyone else. We know gender, age, location, and it’s real data as opposed to the stuff other people 
infer,” Sandberg said. Facebook would learn to track, scrape, store, and analyze UPI to fabricate its own 
targeting algorithms, and like Google it would not restrict extraction operations to what people 
voluntarily shared with the company. Sandberg understood that through the artful manipulation of 
Facebook’s culture of intimacy and sharing, it would be possible to use behavioral surplus not only to 
satisfy demand but also to create demand. For starters, that meant inserting advertisers into the fabric of 
Facebook’s online culture, where they could “invite” users into a “conversation.” 


VIII. Summarizing the Logic and Operations of Surveillance Capitalism 


With Google in the lead, surveillance capitalism rapidly became the default model of information 
capitalism on the web and, as we shall see in coming chapters, gradually drew competitors from every 
sector. This new market form declares that serving the genuine needs of people is less lucrative, and 
therefore less important, than selling predictions of their behavior. Google discovered that we are less 
valuable than others’ bets on our future behavior. This changed everything. 

Behavioral surplus defines Google’s earnings success. In 2016, 89 percent of the revenues of its 


parent company, Alphabet, derived from Google’s targeted advertising programs.®? The scale of raw- 
material flows is reflected in Google’s domination of the internet, processing over 40,000 search queries 
every second on average: more than 3.5 billion searches per day and 1.2 trillion searches per year 
worldwide in 2017.2 

On the strength of its unprecedented inventions, Google’s $400 billion market value edged out 
ExxonMobil for the number-two spot in market capitalization in 2014, only sixteen years after its 
founding, making it the second-richest company in the world behind Apple.2! By 2016, Alphabet/Google 
occasionally wrested the number-one position from Apple and was ranked number two globally as of 
September 20, 2017.2 

It is useful to stand back from this complexity to grasp the overall pattern and how the puzzle pieces fit 
together: 

1. The logic: Google and other surveillance platforms are sometimes described as “two-sided” or 
“multi-sided” markets, but the mechanisms of surveillance capitalism suggest something different.2° 
Google had discovered a way to translate its nonmarket interactions with users into surplus raw material 
for the fabrication of products aimed at genuine market transactions with its real customers: advertisers.“ 
The translation of behavioral surplus from outside to inside the market finally enabled Google to convert 
investment into revenue. The corporation thus created out of thin air and at zero marginal cost an asset 
class of vital raw materials derived from users’ nonmarket online behavior. At first those raw materials 
were simply “found,” a by-product of users’ search actions. Later those assets were hunted aggressively 
and procured largely through surveillance. The corporation simultaneously created a new kind of 
marketplace in which its proprietary “prediction products” manufactured from these raw materials could 
be bought and sold. 

The summary of these developments is that the behavioral surplus upon which Google’s fortune rests 
can be considered as surveillance assets. These assets are critical raw materials in the pursuit of 
surveillance revenues and their translation into surveillance capital. The entire logic of this capital 
accumulation is most accurately understood as surveillance capitalism, which is the foundational 
framework for a surveillance-based economic order: a surveillance economy. The big pattern here is one 
of subordination and hierarchy, in which earlier reciprocities between the firm and its users are 
subordinated to the derivative project of our behavioral surplus captured for others’ aims. We are no 
longer the subjects of value realization. Nor are we, as some have insisted, the “product” of Google’s 
sales. Instead, we are the objects from which raw materials are extracted and expropriated for Google’s 
prediction factories. Predictions about our behavior are Google’s products, and they are sold to its actual 
customers but not to us. We are the means to others’ ends. 

Industrial capitalism transformed nature’s raw materials into commodities, and surveillance 
capitalism lays its claims to the stuff of human nature for a new commodity invention. Now it is human 
nature that is scraped, torn, and taken for another century’s market project. It is obscene to suppose that 
this harm can be reduced to the obvious fact that users receive no fee for the raw material they supply. 
That critique is a feat of misdirection that would use a pricing mechanism to institutionalize and therefore 
legitimate the extraction of human behavior for manufacturing and sale. It ignores the key point that the 
essence of the exploitation here is the rendering of our lives as behavioral data for the sake of others’ 
improved control of us. The remarkable questions here concern the facts that our lives are rendered as 
behavioral data in the first place; that ignorance is a condition of this ubiquitous rendition; that decision 
rights vanish before one even knows that there is a decision to make; that there are consequences to this 
diminishment of rights that we can neither see nor foretell; that there is no exit, no voice, and no loyalty, 
only helplessness, resignation, and psychic numbing; and that encryption is the only positive action left to 


discuss when we sit around the dinner table and casually ponder how to hide from the forces that hide 
from us. 

2. The means of production: Google’s internet-age manufacturing process is a critical component of 
the unprecedented. Its specific technologies and techniques, which I summarize as “machine intelligence,” 
are constantly evolving, and it is easy to be intimidated by their complexity. The same term may mean one 
thing today and something very different in one year or in five years. For example, Google has been 
described as developing and deploying “artificial intelligence” since at least 2003, but the term itself is a 
moving target, as capabilities have evolved from primitive programs that can play tic-tac-toe to systems 
that can operate whole fleets of driverless cars. 

Google’s machine intelligence capabilities feed on behavioral surplus, and the more surplus they 
consume, the more accurate the prediction products that result. Wired magazine’s founding editor, Kevin 
Kelly, once suggested that although it seems like Google is committed to developing its artificial 
intelligence capabilities to improve Search, it’s more likely that Google develops Search as a means of 
continuously training its evolving AI capabilities. This is the essence of the machine intelligence 
project. As the ultimate tapeworm, the machine’s intelligence depends upon how much data it eats. In this 
important respect the new means of production differs fundamentally from the industrial model, in which 
there is a tension between quantity and quality. Machine intelligence is the synthesis of this tension, for it 
reaches its full potential for quality only as it approximates totality. 

As more companies chase Google-style surveillance profits, a significant fraction of global genius in 
data science and related fields is dedicated to the fabrication of prediction products that increase click- 
through rates for targeted advertising. For example, Chinese researchers employed by Microsoft’s Bing’s 
research unit in Beijing published breakthrough findings in 2017. “Accurately estimating the click-through 
rate (CTR) of ads has a vital impact on the revenue of search businesses; even a 0.1% accuracy 
improvement in our production would yield hundreds of millions of dollars in additional earnings,” they 
begin. They go on to demonstrate a new application of advanced neural networks that promises 0.9 
percent improvement on one measure of identification and “significant click yield gains in online 
traffic.”2° Similarly, a team of Google researchers introduced a new deep-neural network model, all for 
the sake of capturing “predictive feature interactions” and delivering “state-of-the-art performance” to 
improve click-through rates. Thousands of contributions like these, some incremental and some 
dramatic, equate to an expensive, sophisticated, opaque, and exclusive twenty-first-century “means of 
production.” 

3. The products: Machine intelligence processes behavioral surplus into prediction products 
designed to forecast what we will feel, think, and do: now, soon, and later. These methodologies are 
among Google’s most closely guarded secrets. The nature of its products explains why Google repeatedly 
claims that it does not sell personal data. What? Never! Google executives like to claim their privacy 
purity because they do not sell their raw material. Instead, the company sells the predictions that only it 
can fabricate from its world-historic private hoard of behavioral surplus. 

Prediction products reduce risks for customers, advising them where and when to place their bets. The 
quality and competitiveness of the product are a function of its approximation to certainty: the more 
predictive the product, the lower the risks for buyers and the greater the volume of sales. Google has 
learned to be a data-based fortune-teller that replaces intuition with science at scale in order to tell and 
sell our fortunes for profit to its customers, but not to us. Early on, Google’s prediction products were 
largely aimed at sales of targeted advertising, but as we shall see, advertising was the beginning of the 
surveillance project, not the end. 

4. The marketplace: Prediction products are sold into a new kind of market that trades exclusively in 


future behavior. Surveillance capitalism’s profits derive primarily from these behavioral futures 
markets. Although advertisers were the dominant players in the early history of this new kind of 
marketplace, there is no reason why such markets are limited to this group. The new prediction systems 
are only incidentally about ads, in the same way that Ford’s new system of mass production was only 
incidentally about automobiles. In both cases the systems can be applied to many other domains. The 
already visible trend, as we shall see in the coming chapters, is that any actor with an interest in 
purchasing probabilistic information about our behavior and/or influencing future behavior can pay to 
play in markets where the behavioral fortunes of individuals, groups, bodies, and things are told and sold 


(see Figure 2). 


H Surveillance capitalism begins with the discovery of 
The D ISCOVEe ry behavioral surplus. More behavioral data are 
: rendered than required for service improvements. 
of Behavi O ral This surplus feeds machine intelligence - the new 
means of production - that fabricates predictions of 
S Uu rp | US user behavior. These products are sold to business 
customers in new behavioral futures markets. The 
Behavioral Value Reinvestment Cycle is 
subordinated to this new logic. 


Markets Prediction 
in Future Products 
Behavior 


a WMS Y, 
~~ SURPLUS - 
OL | 1\ SB 


Rendered 
Surveillance Behavior 


Revenues Neansak 


Production 


Analytics 


Service 


Shoshana Zuboff Im provements 


The Age of Surveillance Capitalism 


Figure 2: The Discovery of Behavioral Surplus 


CHAPTER FOUR 


THE MOAT AROUND THE CASTLE 


The hour of birth their only time in college, 
They were content with their precocious knowledge, 
To know their station and be right forever. 


—W. H. AUDEN 
SONNETS FROM CHINA, I 


I. Human Natural Resources 


Google’s former CEO Eric Schmidt credits Hal Varian’s early examination of the firm’s ad auctions with 
providing the eureka moment that clarified the true nature of Google’s business: “All of a sudden, we 
realized we were in the auction business.”! Larry Page is credited with a very different and far more 
profound answer to the question “What is Google?” Douglas Edwards recounts a 2001 session with the 
founders that probed their answers to that precise query. It was Page who ruminated, “If we did have a 
category, it would be personal information.... The places you’ve seen. Communications.... Sensors are 
really cheap.... Storage is cheap. Cameras are cheap. People will generate enormous amounts of data.... 
Everything you’ve ever heard or seen or experienced will become searchable. Your whole life will be 
searchable.”2 

Page’s vision perfectly reflects the history of capitalism, marked by taking things that live outside the 
market sphere and declaring their new life as market commodities. In historian Karl Polanyi’s 1944 grand 
narrative of the “great transformation” to a self-regulating market economy, he described the origins of 
this translation process in three astonishing and crucial mental inventions that he called “commodity 
fictions.” The first was that human life could be subordinated to market dynamics and reborn as “labor” to 
be bought and sold. The second was that nature could be translated into the market and reborn as “land” 
or “real estate.” The third was that exchange could be reborn as “money.”? Nearly eighty years earlier, 
Karl Marx had described the taking of lands and natural resources as the original “big bang” that ignited 
modern capital formation, calling it “primitive accumulation.”4 

The philosopher Hannah Arendt complicated both Polanyi’s and Marx’s notion. She observed that 
primitive accumulation wasn’t just a one-time primal explosion that gave birth to capitalism. Rather, it is 
a recurring phase in a repeating cycle as more aspects of the social and natural world are subordinated to 
the market dynamic. Marx’s “original sin of simple robbery,” she wrote, “had eventually to be repeated 


lest the motor of capital accumulation suddenly die down.”2 


In our time of pro-market ideology and practice, this cycle has become so pervasive that we 
eventually fail to notice its audacity or contest its claims. For example, you can now “purchase” human 
blood and organs, someone to have your baby or stand in line for you or hold a public parking space, a 
person to comfort you in your grief, and the right to kill an endangered animal. The list grows longer each 
day.® 

Social theorist David Harvey builds on Arendt’s insight with his notion of “accumulation by 
dispossession”: “What accumulation by dispossession does is to release a set of assets... at very low 
(and in some instances zero) cost. Overaccumulated capital can seize hold of such assets and immediately 
turn them to profitable use.” He adds that entrepreneurs who are determined to “join the system” and 
enjoy “the benefits of capital accumulation” are often the ones who drive this process of dispossession 
into new, undefended territories.2 

Page grasped that human experience could be Google’s virgin wood, that it could be extracted at no 
extra cost online and at very low cost out in the real world, where “sensors are really cheap.” Once 
extracted, it is rendered as behavioral data, producing a surplus that forms the basis of a wholly new 
class of market exchange. Surveillance capitalism originates in this act of digital dispossession, brought 
to life by the impatience of over-accumulated investment and two entrepreneurs who wanted to join the 
system. This is the lever that moved Google’s world and shifted it toward profit. 

Today’s owners of surveillance capital have declared a fourth fictional commodity expropriated from 
the experiential realities of human beings whose bodies, thoughts, and feelings are as virgin and 
blameless as nature’s once-plentiful meadows and forests before they fell to the market dynamic. In this 
new logic, human experience is subjugated to surveillance capitalism’s market mechanisms and reborn 
as “behavior.” These behaviors are rendered into data, ready to take their place in a numberless queue 
that feeds the machines for fabrication into predictions and eventual exchange in the new behavioral 
futures markets. 

The commodification of behavior under surveillance capitalism pivots us toward a societal future in 
which market power is protected by moats of secrecy, indecipherability, and expertise. Even when 
knowledge derived from our behavior is fed back to us as a quid pro quo for participation, as in the case 
of so-called “personalization,” parallel secret operations pursue the conversion of surplus into sales that 
point far beyond our interests. We have no formal control because we are not essential to this market 
action. 

In this future we are exiles from our own behavior, denied access to or control over knowledge 
derived from its dispossession by others for others. Knowledge, authority, and power rest with 
surveillance capital, for which we are merely “human natural resources.” We are the native peoples now 
whose tacit claims to self-determination have vanished from the maps of our own experience. 

Digital dispossession is not an episode but a continuous coordination of action, material, and 
technique, not a wave but the tide itself. Google’s leaders understood from the start that their success 
would require continuous and pervasive fortifications designed to defend their “repetitive sin’ from 
contest and constraint. They did not want to be bound by the disciplines typically imposed by the private 
market realm of corporate governance or the democratic realm of law. In order for them to assert and 
exploit their freedom, democracy would have to be kept at bay. 

“How did they get away with it?” It is an important question that we will return to throughout this 
book. One set of answers depends on understanding the conditions of existence that create and sustain 
demand for surveillance capitalism’s services. This theme was summarized in Chapter 2’s discussion of 
the “collision.” A second set of answers depends upon a clear grasp of surveillance capitalism’s basic 


mechanisms and laws of motion. This exploration has begun and will continue through Part II. 

A third set of answers requires an appreciation of the political and cultural circumstances and 
strategies that advanced surveillance capitalism’s claims and protected them from fatal challenge. It is 
this third domain that we pursue in the sections that follow. No single element is likely to have done the 
job, but together a convergence of political circumstances and proactive strategies helped enrich the 
habitat in which this mutation could root and flourish. These include (1) the relentless pursuit and defense 
of the founders’ “freedom” through corporate control and an insistence on the right to lawless space; (2) 
the shelter of specific historical circumstances, including the policies and juridical orientation of the 
neoliberal paradigm and the state’s urgent interest in the emerging capabilities of behavioral surplus 
analysis and prediction in the aftermath of the September 2001 terror attacks; and (3) the intentional 
construction of fortifications in the worlds of politics and culture, designed to protect the kingdom and 
deflect any close scrutiny of its practices. 


IT. The Cry Freedom Strategy 


One way that Google’s founders institutionalized their freedom was through an unusual structure of 
corporate governance that gave them absolute control over their company. Page and Brin were the first to 
introduce a dual-class share structure to the tech sector with Google’s 2004 public offering. The two 
would control the super-class “B” voting stock, shares that each carried ten votes, as compared to the “A” 
class of shares, which each carried only one vote. This arrangement inoculated Page and Brin from 
market and investor pressures, as Page wrote in the “Founder’s Letter” issued with the IPO: “In the 
transition to public ownership, we have set up a corporate structure that will make it harder for outside 
parties to take over or influence Google.... The main effect of this structure is likely to leave our team, 
especially Sergey and me, with increasingly significant control over the company’s decisions and fate, as 
Google shares change hands.”2 

In the absence of standard checks and balances, the public was asked to simply “trust” the founders. 
Schmidt would voice this theme on their behalf whenever challenged on the subject. For example, at the 
Cato Institute in December 2014, Schmidt was asked about the possibility of abuse of power at Google. 
He simply assured the audience of the continuity of the firm’s dynastic line. Page had succeeded Schmidt 
as CEO in 2011, and the current leaders would handpick future leaders: “We’re fine with Larry... same 
circus, same clowns... it’s the same people... all of us who built Google have the same view, and I am 
sure our successors will have the same view.”2 

By that year, Page and Brin had a 56 percent majority vote, which they used to impose a new tri-class 
share structure, adding a “C” class of zero-voting-rights stock.12 As Bloomberg Businessweek observed, 
“The neutered ‘C’ shares ensure Page and Brin retain control far into the future....”"! By 2017, Brin and 
Page controlled 83 percent of the super-voting-class “B” shares, which translated into 51 percent of the 
voting power."2 

Many Silicon Valley founders followed Google’s lead. By 2015, 15 percent of IPOs were introduced 
with a dual-class structure, compared to 1 percent in 2005, and more than half of those were for 
technology companies./? Most significantly, Facebook’s 2012 IPO featured a two-tiered stock structure 
that left founder Mark Zuckerberg in control of voting rights. The company then issued nonvoting class 
“C” shares in 2016, solidifying Zuckerberg’s personal control over every decision.4 

While financial scholars and investors debated the consequences of these share structures, absolute 
corporate control enabled the Google and Facebook founders to aggressively pursue acquisitions, 


establishing an arms race in two critical arenas. State-of-the-art manufacturing depended on machine 
intelligence, compelling Google and later Facebook to acquire companies and talent representing its 
disciplines: facial recognition, “deep learning,” augmented reality, and more.1° But machines are only as 
smart as the volume of their diet allows. Thus, Google and Facebook vied to become the ubiquitous net 
positioned to capture the swarming schools of behavioral surplus flowing from every computer-mediated 
direction. To this end the founders paid outsized premiums for the chance to corner behavioral surplus 
through acquisitions of an ever-expanding roster of key supply routes. 

In 2006, for example, just two years after its IPO, Google paid $1.65 billion for a one-and-a-half- 
year-old startup that had never made any money and was besieged by copyright-infringement lawsuits: 
YouTube. While the move was called “crazy” and the company was criticized for the outsized price tag, 
Schmidt went on the offensive, freely admitting that Google had paid a $1 billion premium for the video- 
sharing site, though saying little about why. By 2009, a canny Forrester Research media analyst had 
unpacked the mystery: “It actually becomes worth the additional value because Google can tie all of its 
advertising expertise and search traffic into YouTube... it ensures that these millions and millions of 
viewers are coming to a Google-owned site rather than someone’s else’s site.... As a loss leader goes, if 
it never makes its money back, it’s still going to be worth it.” 

Facebook’s Zuckerberg pursued the same strategies, paying “astronomical” prices for a “fast and 
furious” parade of typically unprofitable startups like virtual reality firm Oculus ($2 billion) and the 
messaging application WhatsApp ($19 billion), thus ensuring Facebook’s ownership of the gargantuan 
flows of human behavior that would pour through these pipes. Consistent with the extraction imperative, 
Zuckerberg told investors that he would not consider driving revenue until the service reaches “billions” 
of users./2 As one tech journalist put it, “There’s no real need for Zuckerberg to chat with the board... 
there’s no way for shareholders to check Zuckerberg’s antics....”12 

It’s worth noting that an understanding of this logic of accumulation would have usefully contributed to 
the EU Commission’s deliberations on the WhatsApp acquisition, which was permitted based on 
assurances that data flows from the two businesses would remain separate. The commission would 
discover later that the extraction imperative and its necessary of economies of scale in supply operations 
compel the integration of surplus flows in the quest for better prediction products.”2 

Google’s founders constructed a corporate form that gave them absolute control in the market sphere, 
and they also pursued freedom in the public sphere. A key element of Google’s freedom strategy was its 
ability to discern, construct, and stake its claim to unprecedented social territories that were not yet 
subject to law. Cyberspace is an important character in this drama, celebrated on the first page of Eric 
Schmidt and Jared Cohen’s book on the digital age: “The online world is not truly bound by terrestrial 
laws... it’s the world’s largest ungoverned space.”2! They celebrate their claim to operational spaces 
beyond the reach of political institutions: the twenty-first-century equivalent of the “dark continents” that 
drew nineteenth-century European speculators to their shores. 

Hannah Arendt’s examination of British capitalists’ export of over-accumulated capital to Asia and 
Africa in the mid-nineteenth century helps to develop this analogy: “Here, in backward regions without 
industries and political organizations, where violence was given more latitude than in any Western 
country, the so-called laws of capitalism were actually allowed to create realities.... The secret of the 
new happy fulfillment was precisely that economic laws no longer stood in the way of the greed of the 
owning classes.” 

This kind of lawlessness has been a critical success factor in the short history of surveillance 
capitalism. Schmidt, Brin, and Page have ardently defended their right to freedom from law even as 
Google grew to become what is arguably the world’s most powerful corporation.*? Their efforts have 


been marked by a few consistent themes: that technology companies such as Google move faster than the 
state’s ability to understand or follow, that any attempts to intervene or constrain are therefore fated to be 
ill-conceived and stupid, that regulation is always a negative force that impedes innovation and progress, 
and that lawlessness is the necessary context for “technological innovation.” 

Schmidt, Page, and Brin have each been outspoken on these themes. In a 2010 interview with the Wall 
Street Journal, Schmidt insisted that Google needed no regulation because of strong incentives to “treat 
its users right.”24 In 2011 Schmidt cited former Intel CEO Andy Grove’s antidemocratic formula to a 
Washington Post reporter, commenting that Grove’s idea “works for me.” Google was determined to 
protect itself from the slow pace of democratic institutions: 


This is an Andy Grove formula.... “High tech runs three-times faster than normal businesses. And the government runs three-times 
slower than normal businesses. So we have a nine-times gap.... And so what you want to do is you want to make sure that the 


government does not get in the way and slow things down. »2d 


Business Insider covered Schmidt’s remarks at the Mobile World Congress that same year, writing, 
“When asked about government regulation, Schmidt said that technology moves so fast that governments 
really shouldn’t try to regulate it because it will change too fast, and any problem will be solved by 
technology. ‘We’ll move much faster than any government.’”2° 

Both Brin and Page are even more candid in their contempt for law and regulation. CEO Page 
surprised a convocation of developers in 2013 by responding to questions from the audience, commenting 
on the “negativity” that hampered the firm’s freedom to “build really great things” and create 
“interoperable” technologies with other companies: “Old institutions like the law and so on aren’t 
keeping up with the rate of change that we’ve caused through technology.... The laws when we went 
public were 50 years old. A law can’t be right if it’s 50 years old, like it’s before the internet.” When 
asked his thoughts on how to limit “negativity” and increase “positivity,” Page reflected, “Maybe we 
should set aside a small part of the world... as technologists we should have some safe places where we 
can try out some new things and figure out what is the effect on society, what’s the effect on people, 
without having to deploy kind of into the normal world.”2/ 

It is important to understand that surveillance capitalists are impelled to pursue lawlessness by the 
logic of their own creation. Google and Facebook vigorously lobby to kill online privacy protection, limit 
regulations, weaken or block privacy-enhancing legislation, and thwart every attempt to circumscribe 
their practices because such laws are existential threats to the frictionless flow of behavioral surplus. 

Extraction quarry must be both unprotected and available at zero cost if this logic of accumulation is to 
succeed. These requirements are also an Achilles heel. Code is law for Google now, but the risk of new 
laws in its established and anticipated territories remains a persistent danger to surveillance capitalism. If 
new laws were to outlaw extraction operations, the surveillance model would implode. This market form 
must either gird itself for perpetual conflict with the democratic process or find new ways to infiltrate, 
seduce, and bend democracy to its ends if it is to fulfill its own inner logic. The survival and success of 
surveillance capitalism depend upon engineering collective agreement through all available means while 
simultaneously ignoring, evading, contesting, reshaping, or otherwise vanquishing laws that threaten free 
behavioral surplus. 

These claims to lawless space are remarkably similar to those of the robber barons of an earlier 
century. Like the men at Google, the late-nineteenth-century titans claimed undefended territory for their 
own interests, declared the righteousness of their self-authorizing prerogatives, and defended their new 
capitalism from democracy at any cost. At least in the US case, we have been here before. 


Economic historians describe the dedication to lawlessness among the Gilded Age “robber barons” 
for whom Herbert Spencer’s social Darwinism played the same role that Hayek, Jensen, and even Ayn 
Rand play for today’s digital barons. In the same way that surveillance capitalists excuse their 
corporations’ unprecedented concentrations of information and wealth as the unavoidable result of 
“network effects” and “winner-take-all” markets, the Gilded Age industrialists cited Spencer’s specious, 
pseudoscientific “survival of the fittest” as proof of a divine plan intended to put society’s wealth in the 
hands of its most aggressively competitive individuals.72 

The Gilded Age millionaires, like today’s surveillance capitalists, stood on the frontier of a vast 
discontinuity in the means of production with nothing but blank territory in which to invent a new 
industrial capitalism free from constraints on the use of labor, the nature of working conditions, the extent 
of environmental destruction, the sourcing of raw materials, or even the quality of their own products. 
And like their twenty-first-century counterparts, they did not hesitate to exploit the very law that they 
despised, flying the banner of “private property” and “freedom of contract,” much as surveillance 
capitalists march under the flag of freedom of speech as the justification for unobstructed technological 
“progress,” a topic to which we shall return. 

Imbued with the conviction that “the state had neither right nor reason to interfere in the workings of 
the economy,” the Gilded Age millionaires joined forces to defend the “rights of capital” and limit the 
role of elected representatives in setting policy or developing legislation.*2 There was no need for law, 
they argued, when one had the “law of evolution,” the “laws of capital,” and the “laws of industrial 
society.” John Rockefeller insisted that his outsized oil fortune was the result of “the natural law of trade 
development.” Jay Gould, when questioned by Congress on the need for federal regulation of railroad 
rates, replied that rates were already regulated by “the laws of supply and demand, production and 
consumption.”2! The millionaires mobilized in 1896 to defeat the populist Democrat William Jennings 
Bryan, who had vowed to tether economic policy to the political realm, including regulating the railroads 
and protecting the people from “robbery and oppression.”22 

The bottom line for Gilded Age business elites was that the most effective way to protect the original 
sin of that economic era was, as David Nasaw put it, “to circumscribe democracy.” They did this by 
lavishly funding their own political candidates as well as through the careful honing and aggressive 
dissemination of an ideological attack on the very notion of democracy’s right to interfere in the economic 
realm.’ Their industries were to be “self-regulating”: free to follow their own evolutionary laws. 
“Democracy,” they preached, “had its limits, beyond which voters and their elected representatives dared 
not trespass lest economic calamity befall the nation.” In our discussion of “fortifications” we’ ll see that 
Google revived all of these strategies and more. But first we explore the unique circumstances that 
sheltered the young company and protected its discovery of human experience as a limitless resource ripe 
for the taking. 


III. Shelter: The Neoliberal Legacy 


Google’s leaders were also favored by historical circumstance. Both Google and the wider surveillance 
capitalist project were the beneficiaries of two developments that contributed to a uniquely sheltering 
habitat for the surveillance mutation. The first is the neoliberal capture of the governmental machinery for 
oversight and regulation of the US economy, the framework of which we discussed in Chapter 2.2 

A fascinating study by University of California law professor Jodi Short empirically illustrates the 


role of neoliberal ideology as one important explanation for Google’s ambitions and successful defense of 


lawless territory.°° Short analyzed 1,400 law review articles on the subject of regulation, all published 
between 1980 and 2005. As the influence of Hayek and Friedman predicts, the dominant theme of this 
literature was “the coercive nature of administrative government” and the systematic conflation of 
industry regulation with “tyranny” and “authoritarianism.” According to this worldview, all regulation is 
burdensome, and bureaucracy must be repudiated as a form of human domination. Short observes that 
during the sample decades these fears were even more influential in shaping regulatory approaches than 
rational arguments about cost and efficiency, and she identifies two points of origin for these anxieties. 

The first source was in the US business community’s opposition to New Deal reforms, which, not 
unlike the propaganda of the Gilded Age millionaires, cast resistance to regulation as a “righteous fight to 
defend democracy from dictatorship.”2 A second source was the dread of totalitarianism and 
collectivism incited by World War II and the cold war, a direct inheritance from Hayek. These defensive 
themes infiltrated and reshaped US political thought, and they gradually transformed policy makers’ 
assumptions about the regulatory role of the state.=8 

Short found several suggested remedies for “coercive” governmental regulation in the literature, but 
the most salient, especially after 1996—the very years when digital technology and the internet were 
becoming mainstream—was “self-regulation.” The idea here is that firms set their own standards, monitor 
their own compliance with those standards, and even judge their own conduct in order to “voluntarily 
report and remediate violations.”22 By the time of Google’s public offering in 2004, self-regulation was 
fully enshrined within government and across the business community as the single most effective tool for 
regulation without coercion and the antidote to any inclination toward collectivism and the centralization 
of power.2 

This neoliberal legacy was a windfall for the new surveillance capitalists. As another legal scholar, 
Frank Pasquale, observed, it produced a model that treated privacy as a competitive good, assuming that 
“consumers” engage only with services that offer the level of privacy they seek. Regulatory interference, 
according to this view, would only undermine competitive diversity. It also credits the “notice and 
consent” model—click-wrap and its “sadistic” relatives—as accurate signals of individual privacy 
choices.#! 

The neoliberal zeitgeist also favored Google’s leaders, and later their fellow travelers in the 
surveillance project, as they sought shelter for their inventions beneath claims of First Amendment rights 
to freedom of expression. This is a complex and contested arena in which constitutional law and political 
ideology are thoroughly entangled, and I point out just a few elements here in order to better understand 
the habitat that nurtured the new surveillance market form.“ 

The key dynamic here is that First Amendment jurisprudence, especially over the last two decades, 
has reflected a “conservative-libertarian” interpretation of First Amendment rights. As constitutional law 
scholar Steven Heyman suggests, “In recent decades, the First Amendment has become one of the most 
important means by which judges have sought to advance a conservative-libertarian agenda.” This has 
produced many dramatic judicial decisions, including the US Supreme Court’s rejection of any constraints 
on the role of money in election campaigns, its rejection of restrictions on hate speech and pornography, 
and its holding that the right to free association takes precedence over state civil rights laws that bar 
discrimination. 

As many legal scholars observe, the ideological orientation of contemporary First Amendment judicial 
reasoning asserts a close connection between free speech and property rights. The logic that links 
ownership to an absolute entitlement to freedom of expression has led to a privileging of corporate action 
as “speech” deserving of constitutional protection.“ Some scholars regard this as a dangerous reversion 
to the feudal doctrines from which corporate law evolved in the seventeenth century. Those medieval 


legal principles limited the sovereign’s authority over “the corporations of Aristocracy, Church, guilds, 
universities, and cities... who asserted the right to rule themselves.” One result is that US courts have 
been “quick to see the possibilities of governmental overreach, but much less willing to see the problems 
of ‘private,’ let alone corporate, power.” 

In this context, surveillance capitalists vigorously developed a “cyberlibertarian” ideology that Frank 
Pasquale describes as “free speech fundamentalism.” Their legal teams aggressively assert First 
Amendment principles to fend off any form of oversight or externally imposed constraints that either limit 
the content on their platforms or the “algorithmic orderings of information” produced by their machine 
operations.4° As one attorney who has represented many of the leading surveillance capitalists puts it, 
“The lawyers working for these companies have business reasons for supporting free expression. Indeed, 
all of these companies talk about their businesses in the language of free speech.”“2 

This is one respect in which the surveillance capitalists are not unprecedented. Adam Winkler, a 
historian of corporate rights, reminds us, “Throughout American history the nation’s most powerful 
corporations have persistently mobilized to use the Constitution to fight off unwanted government 
regulations.”“8 Although today’s mobilizations are not original, Winkler’s careful account demonstrates 
the effects of past mobilizations on the distribution of power and wealth in US society and the strength of 
democratic values and principles in each era. 

The key point for our story in the age of surveillance capitalism is that the expansion of opportunities 
for free expression associated with the internet has been an emancipatory force in many vital respects, but 
this fact must not blind us to another condition: free speech fundamentalism has deflected careful scrutiny 
of the unprecedented operations that constitute the new market form and account for its spectacular 
success. The Constitution is exploited to shelter a range of novel practices that are antidemocratic in their 
aims and consequences and fundamentally destructive of the enduring First Amendment values intended to 
protect the individual from abusive power. 

In the US, congressional statutes have played an equally or perhaps even more important role in 
sheltering surveillance capitalism from scrutiny. The most celebrated of these is a legislative statute 
known as Section 230 of the Communications Decency Act of 1996, which shields website owners from 
lawsuits and state prosecution for user-generated content. “No provider or user of an interactive computer 
service,” the statute reads, “shall be treated as the publisher or speaker of any information provided by 
another information content provider.”“2 This is the regulatory framework that enables a site such as 
TripAdvisor to include negative hotel reviews and permits Twitter’s aggressive trolls to roam free 
without either company being held to the standards of accountability that typically guide news 
organizations. Section 230 institutionalized the idea that websites are not publishers but rather 
“intermediaries.” As one journalist put it, “To sue an online platform over an obscene blog post would be 
like suing the New York Public Library for carrying a copy of Lolita.”=2 As we shall see, this reasoning 
collapses once surveillance capitalism enters the scene. 

Section 230’s hands-off stance toward companies perfectly converged with the reigning ideology and 
practice of “self-regulation,” leaving the internet companies, and eventually the surveillance capitalists 
among them, free to do what they pleased. The statute was crafted in 1995, during the initial phase of the 
public internet. It aimed to clarify intermediaries’ liability for the content on their websites and resolve a 
controversy created by two contradictory court decisions both involving defamatory posts.2! In 1991 a 
court found that CompuServe was not liable for defamation because it had not reviewed the contents of a 
post before it appeared online. The court reasoned that CompuServe was comparable to a public library, 
bookstore, or newsstand: a distributor, not a publisher. 

Four years later, in 1995, an early provider of web services called Prodigy was sued for a defamatory 


anonymous posting on one of its message boards. This time a New York state court came to the opposite 
conclusion. The key problem as the court saw it was that Prodigy had exercised editorial control by 
moderating its message boards. The company established content guidelines and deleted posts that 
violated those standards. The court concluded that Prodigy was a publisher, not merely a distributor, 
because it had taken responsibility for the content on its site. Were the court’s ruling to stand, internet 
companies would face “a paradoxical no-win situation: the more an ISP tried to keep obscene or harmful 
material away from its users, the more it would be liable for that material.”22 Internet companies faced a 
binary choice: “free speech savior or shield for scoundrels?” 

According to Senator Ron Wyden, Section 230 was intended to resolve that contradiction by 
encouraging internet companies to exercise some control over content without the risk of legal sanctions. 
The very first sentence of the statute mentions “protection for ‘good samaritan’ blocking and screening of 
offensive material.”°4 What Wyden and his colleagues could not have anticipated, and still do not grasp, 
is that the logic of this early controversy no longer holds. Neither CompuServe nor Prodigy was a 
surveillance capitalist, but many of today’s internet intermediaries are committed to the pursuit of 
surveillance revenues. 

This fact fundamentally changes the relationship between the company and the content on its platforms, 
and it explains why surveillance capitalists cannot be compared to the New York Public Library as the 
neutral caretaker of Nabokov’s venerated book. Far from it. Under the regime of surveillance capitalism, 
content is a source of behavioral surplus, as is the behavior of the people who provide the content, as are 
their patterns of connection, communication, and mobility, their thoughts and feelings, and the meta-data 
expressed in their emoticons, exclamation points, lists, contractions, and salutations. That book on the 
bookshelf—along with the records of anyone who may have touched it and when, their location, behavior, 
networks, and so on—is now the diamond mine ready for excavation and plunder, to be rendered into 
behavioral data and fed to the machines on their way to product fabrication and sales. Section 230’s 
protection of the “intermediaries” now functions as another bulwark that shelters this extractive 
surveillance capitalist operation from critical examination. 

There is nothing neutral about the surveillance intermediary now, as the extraction imperative and its 
demand for economies of scale in surplus supply mean that the surveillance capitalists must use every 
means to attract a never-ending tide of content to their shores. They no longer merely host content but 
aggressively, secretly, and unilaterally extract value from that content. As we shall see in Chapter 18, 
economic imperatives require them to forgo as few of these raw materials as possible. That means 
moderating only those extremes that threaten the volume and velocity of surplus by repelling users or 
attracting regulatory scrutiny. This is the reason that firms such as Facebook, Google, and Twitter have 
been reluctant to remove even the most egregious content from their landscapes, and it helps to explain 
why “lawyers for tech companies litigate ferociously to prevent even a sliver of erosion” in Section 
230.22 A statute once crafted to nurture an important new technological milieu is now the legal bulwark 
that protects the asymmetric wealth, knowledge, and power of a rogue capitalism. 


IV. Shelter: Surveillance Exceptionalism 


In his book Surveillance After September 11, surveillance scholar David Lyon writes that in the aftermath 
of the attacks that day, existing surveillance practices were intensified and previous limits were lifted: 
“After several decades in which data-protection officials, privacy watchdogs, civil rights groups, and 
others have tried to mitigate negative social effects of surveillance, we are witnessing a sharp tilt toward 


more exclusionary and intrusive surveillance practices.”=° This abrupt refocusing of governmental power 


and policy after the 9/11 attacks in New York City and Washington, DC, is a second historical condition 
that lent shelter to the fledgling market form. 

Lyon’s characterization is accurate.’ In the years before 9/11, the Federal Trade Commission emerged 
as the key actor defining the debate on internet privacy in the US. For reasons that we have already 
reviewed, the FTC favored self-regulation, and it cajoled internet companies to establish codes of 
conduct, privacy policies, and methods of enforcement.22 But the FTC eventually concluded that self- 
regulation would not be sufficient to protect individual consumers’ privacy on the web. In 2000, still one 
year before the 9/11 attacks, Google’s discovery of behavioral surplus, or the success of AdWords, a 
majority of FTC commissioners issued a report in which they recommended legislation to regulate online 
privacy: “Because self-regulatory initiatives to date fall far short of broad-based implementation of self- 
regulatory programs,” they wrote, “the Commission has concluded that such efforts alone cannot ensure 
that the online marketplace as a whole will follow the standards adopted by industry leaders... 
notwithstanding several years of industry and governmental effort.” The reported noted that a mere 8 
percent of popular websites featured a seal of approval from one of the industry privacy watchdogs.” 

The commissioners proceeded to outline federal legislation that would have protected consumers 
online despite the dominant bias against regulation and in favor of treating internet operations as free 
speech. The recommendations demanded “clear and conspicuous” notice of information practices; 
consumer choice over how personal information is used; access to all personal information, including 
rights to correct or delete; and enhanced security of personal information. Had these been translated into 
law, it is quite possible that many of the foundational elements of surveillance capitalism would have 
been either plainly illegal or at least subject to public examination and contest. 

The FTC effort was short-lived. According to Peter Swire, chief counselor for Privacy in the Clinton 
Administration and later a member of President Obama’s Review Group on Intelligence and 
Communication Technologies, “With the attacks of September 11, 2001, everything changed. The new 
focus was overwhelmingly on security rather than privacy.”©! The privacy provisions debated just months 
earlier vanished from the conversation more or less overnight. In both the US Congress and across the 
EU, legislation was quickly put in place that decisively expanded surveillance activities. The US 
Congress passed the Patriot Act, created the Terrorist Screening Program, and instituted a host of other 
measures that dramatically increased the warrantless collection of personal information. The events of 
9/11 also triggered a steady stream of legislation that expanded the powers of intelligence and law- 
enforcement agencies across Europe, including Germany (a country that had been highly sensitized to 
surveillance under the hammer of both Nazi and Stalinist totalitarianism), the UK, and France. 

In the US the failure to “connect the dots” on the terrorist attack was a source of shame and dismay that 
overwhelmed other concerns. Policy guidelines shifted from “need to know” to “need to share” as 
agencies were urged to tear down walls and blend databases for comprehensive information and 
analysis. In a parallel development, privacy scholar Chris Jay Hoofnagle observes that the threat of 
comprehensive privacy legislation had also mobilized the business community and its lobbyists to either 
“shape or stop” any potential bill. In the post—9/11 political environment, the two forces converged for an 
easy victory.“ 

The internet was the critical target. CIA Director Michael Hayden conceded as much in 2013 when he 
told an audience that in the years following 9/11, the CIA “could be fairly charged with the militarization 
of the world wide web.”© Legislation to regulate online privacy was an immediate casualty. Marc 
Rotenberg, the director of the Electronic Privacy Information Center (EPIC), testified to the 9/11 
Commission on the sudden reversal of privacy concerns, observing that before 9/11, “There was hardly 


any positive discussion about the development of techniques that would enable massive surveillance 
while attempting to safeguard privacy.”©° Swire concurred, noting that as a result of the new emphasis on 
information sharing, “Congress lost interest in regulating information usage in the private sector.... 
Without the threat of legislation, the energy went out of many of the self-regulatory efforts that industry had 
created.”®/ At the FTC, the focus shifted from the broader concerns of privacy rights to a more politically 
palatable “harms-based” strategy, pursuing cases in which concrete physical harms or economic injuries 
could be defined, such as identify theft or database security. 

With legislation off the table, other forces shaped the political environment in which surveillance 
capitalism would root and grow. The 9/11 terrorist attacks thrust the intelligence community into an 
unfamiliar demand curve that insisted on exponential increases in velocity. For all its secrecy, even the 
NSA was subject to the temporalities and legal restrictions of a democratic state. The tempos of 
democracy are slow by design, weighted by redundancies, checks and balances, laws and rules. The 
agencies sought methods of deployment that could rapidly bypass legal and bureaucratic restrictions. 

In this environment of trauma and anxiety, a “state of exception” was invoked to legitimate a new 
imperative: speed at any cost. As Lyon put it, “What 9/11 did was to produce socially negative 
consequences that hitherto were the stuff of repressive regimes and dystopian novels.... The suspension 
of normal conditions is justified with reference to the ‘war on terrorism.’”®? Critical to our story is the 
fact that this state of exception favored Google’s growth and the successful elaboration of its 
surveillance-based logic of accumulation. 

Google’s mission was to “organize and make accessible the world’s information,” and by late 2001 
the intelligence community established “information dominance” in the public’s house, quickly 
institutionalizing it in hundreds of billions of dollars’ worth of state-sponsored global technology 
infrastructure, personnel, and practice. The contours of a new interdependency between public and 
private agents of information dominance began to emerge, one that is best understood through the lens of 
what the sociologist Max Weber once called an “elective affinity’ born of a mutual magnetism that 
originates in shared meanings, interests, and reciprocities. 2 

The elective affinity between public intelligence agencies and the fledgling surveillance capitalist 
Google blossomed in the heat of emergency to produce a unique historical deformity: surveillance 
exceptionalism. The 9/11 attacks transformed the government’s interest in Google, as practices that just 
hours earlier were careening toward legislative action were quickly recast as mission-critical 
necessities. Both institutions craved certainty and were determined to fulfill that craving in their 
respective domains at any price. These elective affinities sustained surveillance exceptionalism and 
contributed to the fertile habitat in which the surveillance capitalism mutation would be nurtured to 
prosperity. 

The elective affinity between public and private missions was evident as early as 2002, when former 
NSA Chief Admiral John Poindexter proposed his Total Information Awareness (TIA) program with a 
vision that reads like an early guide to the foundational mechanisms of behavioral surplus capture and 
analysis: 


If terrorist organizations are going to plan and execute attacks against the United States, their people must engage in transactions and 
they will leave signatures in this information space.... We must be able to pick this signal out of the noise... the relevant information 


extracted from this data must be made available in large-scale repositories with enhanced semantic content for analysis to accomplish 


this task.2 


As CIA Director George Tenet had declared in 1997, “The CIA needs to swim in the Valley,” referring 


to the need to master the new technologies flowing from Silicon Valley.4 In 1999 it opened a CIA-funded 
venture firm in the valley, In-Q-Tel, as a conduit for cutting-edge technologies. The operation was meant 
to be an agency experiment, but after 9/11 it became a critical source of new capabilities and 
relationships, including with Google. As Silicon Valley’s Mercury News reported, “There’s a new 
urgency with the CIA to find technology that makes sense of all the unstructured data floating around on 
the internet and elsewhere. The agency can’t train analysts quickly enough.” In-Q-Tel’s CEO described 
the government’s agencies as “scrambling” and noted that “we’re ina state of hyperactivity now.” 

Surveillance exceptionalism thrived in that hyperactivity. Poindexter’s Total Information Awareness 
program did not obtain congressional support, but an analysis in the MIT Technology Review showed that 
many of the TIA initiatives were quietly reassigned to the Pentagon’s Advanced Research and 
Development Activity (ARDA), which in 2002 received $64 million to fund a research program in “novel 
intelligence from massive data.” In 2004 the US General Accounting Office surveyed 199 data-mining 
projects across dozens of federal agencies and more than 120 programs developed to collect and analyze 
personal data to predict individual behavior.“ The New York Times reported in 2006 that the intelligence 
agencies, backed by a $40 billion annual budget, regularly fielded secretive shopping expeditions to 
Silicon Valley in search of new data-mining and analysis technologies.” 

State security agencies sought ways to avail themselves of Google’s rapidly developing capabilities 
and simultaneously use Google to further develop, commercialize, and diffuse security and surveillance 
technologies with proven intelligence value. If TIA could not be fully developed and integrated in 
Washington, parts of the job could be delegated to Silicon Valley and its standout in information 
dominance: Google. By late summer 2003, Google was awarded a $2.07 million contract to outfit the 
agency with Google search technology. According to documents obtained by Consumer Watchdog under 
the Freedom of Information Act, the NSA paid Google for a “search appliance capable of searching 15 
million documents in twenty-four languages.” Google extended its services for another year at no cost in 
April 2004.2 

In 2003 Google also began customizing its search engine under special contract with the CIA for its 
Intelink Management Office, “overseeing top-secret, secret and sensitive but unclassified intranets for 
CIA and other IC agencies.” Key agencies used Google systems to support an internal wiki called 
Intellipedia that allowed agents to share information across organizations as quickly as it was vacuumed 
up by the new systems.2 In 2004 Google acquired Keyhole, a satellite mapping company founded by John 
Hanke, whose key venture backer was the CIA venture firm, In-Q-Tel. Keyhole would become the 
backbone for Google Earth, and Hanke would go on to lead Google Maps, including the controversial 
Street View Project. In 2009 Google Ventures and In-Q-Tel both invested in a Boston-based startup, 
Recorded Future, that monitors every aspect of the web in real time in order to predict future events. 
Wired reported that it was the first time the CIA-backed venture firm and Google had funded the same 
startup and that both firms had seats on Recorded Future’s board of directors. 

In the decade that followed 9/11, surveillance exceptionalism was also expressed in the flattery of 
imitation, as the NSA tried to become more like Google, emulating and internalizing Google’s capabilities 
in a variety of domains. In 2006 General Keith Alexander outlined his vision for a search tool called 
ICREACH that “would allow unprecedented volumes of... metadata to be shared and analyzed across the 
many agencies in the Intelligence Community.” By late 2007, the program was piloted, boosting the 
number of communications events it shared from 50 billion to more than 850 billion. The system was 
designed with a “Google-like” search interface that enabled analysts to run searches against meta-data 
“selectors” and to extract vital behavioral surplus for analyses that could reveal “social networks,” 
“patterns of life,’ and “habits,” and in general “predict future behavior.”®2 In 2007 two NSA analysts 


wrote an internal training manual on how to find information on the internet. It expressed the agency’s 
keen interest in all things Google with a detailed chapter devoted to a deconstruction of Google Search 
and the Google “hacks” that can uncover information not intended for public distribution.®! 

That year, the elective affinities that infused the intelligence community’s interest in Google were also 
highlighted when Google research director and AI expert Peter Norvig presented at a Pentagon Highlands 
Forum meeting: an exclusive networking event where military and intelligence officials commune with 
members of the high-tech industry, elected officials, elite academics, top corporate executives, and 
defense contractors. In 2001 the forum’s director, Richard O’ Neill, described its work to a Harvard 
audience as “an idea engine, so the ideas that emerge from meetings are available for use by decision 
makers as well as by people from the think tanks.”® It was to be a bridge between the government and 
commercial leaders, especially in Silicon Valley.2° According to one highly detailed account by 
investigative journalist Nafeez Ahmed and cited by legal scholar Mary Anne Franks, the forum was both a 
support system and an incubator of Google’s growth, as well as a connecting and convening force for the 
Pentagon, intelligence agencies, and the young company: “The US intelligence community’s incubation of 
Google from inception occurred through a combination of direct sponsorship and informal networks of 
financial influence, themselves closely aligned with Pentagon interests.” Another legal scholar 
described the “collaboration” between Google and the intelligence community, especially the NSA, as 
“unprecedented.” 

During these years, scholars noted the growing interdependencies between the intelligence agencies, 
resentful of constitutional constraints on their prerogatives, and the Silicon Valley firms.®° The agencies 
craved the lawlessness that a firm such as Google enjoyed. In his 2008 essay “The Constitution in the 
National Surveillance State,” law professor Jack Balkin observed that the Constitution inhibits 
government actors from high-velocity pursuit of their surveillance agenda, and this creates incentives for 
the government “to rely on private enterprise to collect and generate information for it.”®/ Balkin noted 
that the Supreme Court has imposed few privacy restrictions on business records and information that 
people give to third parties. E-mail is typically held in private servers, making its protection “limited if 
not nonexistent.” This absence of law made private companies attractive partners for government actors 
bound to democratic constraints. 

The government’s need to evade constitutional oversight, argues legal scholar Jon Michaels, leads to 
secret public-private intelligence collaborations that tend to be “orchestrated around handshakes rather 
than legal formalities, such as search warrants, and may be arranged this way to evade oversight and, at 
times, to defy the law.”88 He observed that intelligence agencies are irresistibly drawn to “and in some 
respects dependent upon” firms’ privately held data resources. 

Both scholars’ observations were confirmed in 2010, when former NSA Director Mike McConnell 
offered another glimpse into the elective affinities between Google and the intelligence community. 
Writing in the Washington Post, McConnell made clear that Google’s surveillance-based operations in 
data capture, extraction, and analysis were both taken for granted and coveted. Here the boundaries of 
private and public melt in the intense heat of new threats and their high-velocity demands that must be met 
in “milliseconds.” In McConnell’s future there is one “seamless” surveillance empire in which the 
requirements of self-preservation leave no opportunity for the amenities of democracy, with its time- 
wasting practices of due process, evidence, warrants, and law. As McConnell insisted, 


An effective partnership with the private sector must be formed so information can move quickly back and forth from public to 
private and classified to unclassified... to protect the nation’s critical infrastructure. Recent reports of possible partnership between 


Google and the government point to the kind of joint efforts—and shared challenges—that we are likely to see in the future... such 


arrangements will muddy the waters between the traditional roles of the government and the private sector.... Cyberspace knows no 


borders, and our defensive efforts must be similarly seamless. 22 


In the final months of the Obama administration, then Secretary of Defense Ash Carter toured Silicon 
Valley, where he announced a new Defense Innovation Advisory Board, meant to formalize a channel 
between the tech executives and the DOD. Carter appointed Schmidt to the new board and tasked him 
with selecting its members. As Wired concluded, “The government needs Silicon Valley more than ever as 
it seeks to defend from security threats in cyberspace.”2! These facts are amply illustrated in a 
comprehensive treatment of “bulk collection” by an international group of scholars and edited by Indiana 
University’s Fred Cate and Berkeley’s James Dempsey. Cate and Dempsey note the “expansive 
aggregation” of personal data in the hands of private companies: “Governments understandably want 
access to this data.... Essentially every government in the world claims the power to compel disclosure 
of this data by the companies that hold it.”2* Had it not been for surveillance exceptionalism, it is possible 
that these data would not even exist, at least not in their current volume and detail. 

Surveillance exceptionalism helped to shape the evolutionary course of information capitalism by 
creating an environment in which Google’s budding surveillance practices were coveted rather than 
contested. Once again, history offers us no control groups, and we cannot know with certainty whether 
information capitalism might have developed in a different direction had it not been for the sudden new 
interest in surveillance capabilities. For now, it appears that one unanticipated consequence of this 
public-private “elective affinity” was that the fledgling practices of surveillance capitalism were allowed 
to root and grow with little regulatory or legislative challenge, emboldening Google’s young leaders to 
insist on lawlessness as a natural right and, in ways that are even more opaque, emboldening the state to 
grant them that freedom. 

Powerful elective affinities favored the acquisition of certainty at any price, and part of that price 
appears to have been the shelter of surveillance capitalism. In the fullness of time, historians will no 
doubt discover the specifics of these relationships and the ways in which Google’s discoveries in the 
capture and use of behavioral surplus were sheltered from scrutiny, at least in part, because of this new 
habitat of militarized demand. 

In the context of new military purpose, the digital capabilities that were aimed toward the advocacy- 
oriented values of the behavioral value reinvestment cycle flowed toward surveillance without 
impediment. Surveillance assets thrived without risk of sanction and attracted surveillance capital. 
Revenue followed. The situation recalls the auto, steel, and machine tool industries at mid-century, when 
military orders kept plants operating at full capacity. In the end, however, this turned out to be more curse 
than blessing. Military demand distorted and suppressed the innovation process and drove a wedge 
between these industries and their civilian customers, leaving them vulnerable to foreign competitors in 
the globalizing markets of the late 1970s and early 1980s.2 

Similarly, under the conditions of surveillance exceptionalism, Google’s leaders were not compelled 
to undertake the arduous and risky work of inventing an exchange-based advocacy-oriented market form 
when the surveillance model was so lucrative. Why risk experimentation with more-organic paths to 
monetization when surveillance and extraction operations were safe from law and hugely profitable? 
Eventually, it wasn’t just Google asking these questions; every other internet business faced the same 
choices. Once surveillance revenues set the bar for venture capitalists and Wall Street analysts, it became 
that much easier for internet companies to go with the flow. Then it became onerous not to. 


V. Fortifications 


Why is it that so many years after the events that triggered the mania for information dominance, 
surveillance capitalism still runs with relatively little impediment, especially in the US? The intervening 
years have seen the proliferation of thousands of institutional facts that normalized surveillance 
capitalism’s practices and made them appear necessary and inevitable: the discovery of behavioral 
surplus and the massive accumulations of capital and material that followed, the proliferation of devices 
and services, the integration of data flows, and the institutionalization of futures markets in human 
behavior. 

This does not mean that we should succumb to the natural fallacy and interpret this flourishing as a 
signal of surveillance capitalism’s inherent worthiness or inevitability. In the coming chapters we will 
uncover many additional factors that have contributed to this success, but here I want to focus on Google’s 
proactive efforts to build fortifications around its supply chains in order to protect surplus flows from 
challenge. 

Although many elements of this fortification strategy have been well publicized, their importance for 
our story lies in the fact that each is one aspect of a multipronged effort that deflects scrutiny from core 
operations in order to maintain the flow of free, unregulated behavioral surplus. Fortifications have been 
erected in four key arenas to protect Google, and eventually other surveillance capitalists, from political 
interference and critique: (1) the demonstration of Google’s unique capabilities as a source of 
competitive advantage in electoral politics; (2) a deliberate blurring of public and private interests 
through relationships and aggressive lobbying activities; (3) a revolving door of personnel who migrated 
between Google and the Obama administration, united by elective affinities during Google’s crucial 
growth years of 2009-2016; and (4) Google’s intentional campaign of influence over academic work and 
the larger cultural conversation so vital to policy formation, public opinion, and political perception. The 
results of these four arenas of defense contribute to an understanding of how surveillance capitalism’s 
facts came to stand and why they continue to thrive. 

First, Google demonstrated that the same predictive knowledge derived from behavioral surplus that 
had made the surveillance capitalists wealthy could also help candidates win elections. To make the 
point, Google was ready to apply its magic to the red-hot core of twenty-first-century campaigning, 
beginning with the 2008 Obama presidential campaign. Schmidt had a leading role in organizing teams 
and guiding the implementation of cutting-edge data strategies that would eclipse the traditional political 
arts with the science of behavioral prediction.“ Indeed, “At Obama’s Chicago headquarters... they 
remodeled the electorate in every battleground state each weekend... field staff could see the events’ 
impact on the projected behaviors and beliefs of every voter nationwide.” 

Research by media scholars Daniel Kreiss and Philip Howard indicates that the 2008 Obama 
campaign compiled significant data on more than 250 million Americans, including “a vast array of online 
behavioral and relational data collected from use of the campaign’s web site and third-party social media 
sites such as Facebook....”2° Journalist Sasha Issenberg, who documented these developments in his book 
The Victory Lab, quotes one of Obama’s 2008 political consultants who likened predictive modeling to 
the tools of a fortune-teller: “We knew who... people were going to vote for before they decided.” 

Obama used his proximity to Schmidt to cement his own identity as the innovation candidate poised to 
disrupt business as usual in Washington.22 Once elected, Schmidt joined the Transition Economic 
Advisory Board and appeared next to Obama at his first postelection press conference.2 According to 
Politico, “The image alone of Schmidt standing elbow-to-elbow with Obama’s top economic thinkers 
was enough to send shivers up the spine of Google’s competitors. ‘This terrifies Microsoft,’ said a 


Democratic lobbyist familiar with the industry. ‘There’s a reason why people are scared to death of 
Google.’”12 

Schmidt’s role in President Obama’s election was but one chapter in a long, and by now fabled, 
relationship that some have described as a “love affair.”2! Not surprisingly, Schmidt took on an even 
more prominent role in the 2012 reelection campaign. He led in fundraising and in breaking new technical 
ground, and he “personally oversaw the voter-turnout system on election night.”1 

Political correspondent Jim Rutenberg’s New York Times account of the data scientists’ seminal role in 
the 2012 Obama victory offers a vivid picture of the capture and analysis of behavioral surplus as a 
political methodology. The campaign knew “every single wavering voter in the country that it needed to 
persuade to vote for Obama, by name, address, race, sex, and income,” and it had figured out how to 
target its television ads to these individuals. One breakthrough was the “persuasion score” that identified 
how easily each undecided voter could be persuaded to vote for the Democratic candidate. 1% 

The facts of behavioral surplus and its predictive power were kept top secret in the Obama 
campaigns, just as they are in Google, Facebook, and other domains of information dominance. As 
Rutenberg observed, “The extent to which the campaign used the newest tech tools to look into people’s 
lives and the sheer amount of personal data its vast servers were crunching remained largely shrouded. 
The secrecy... was partly... to maintain their competitive edge. But it was also no doubt because they 
worried that practices like ‘data mining’ and ‘analytics’ could make voters uncomfortable.” 

Second, with the 2012 election in sight, an interview with the Washington Post in 2011 found Schmidt 
boasting about another fortification strategy: “The staffers are young—the staffers get it.... So that’s what 
we depend on. And of course we’ve hired ex-staffers as well. They all know each other. So that’s how it 
really works.”/2 Google’s political utilities paved the way for the unusually crowded and fast-spinning 
revolving door between the East Coast and West Coast centers of power. The Google Transparency 
Project analyzed the movement of staff between the Googlesphere (the company plus its affiliates and its 
law and lobbying firms) and the government (including the White House, Congress, government agencies, 
federal commissions, and national political campaigns) during the Obama years. It found that by April 
2016, 197 individuals had migrated from the government into the Googlesphere, and 61 had moved in the 
other direction. Among these, 22 White House officials went to work for Google, and 31 Googlesphere 
executives joined the White House or federal advisory boards with direct relevance to Google’s 
business.128 

Third, just to be on the safe side, Google shared its largesse throughout the political system. In 
Schmidt’s 2014 book, coauthored with longtime Google executive Jonathan Rosenberg, the CEO 
aggressively developed the theme of government as the shill of incumbents colluding to inhibit change, 
with Google on the outside: an upstart and a disrupter. The authors voiced their disdain for politicians and 
lobbyists, writing, “This is the natural path of politicians since incumbents tend to have a lot more money 
than disrupters and are quite expert in using it to bend the political will of any democratic 
government.” 122 

That same year, while Schmidt disparaged incumbents and their political sway, Google spent more on 
lobbying than any other corporation—over $17 million and nearly twice as much as surveillance rival 
Facebook. In the next few years, as the White House changed hands, Google maintained its pace, 
outspending every other company with a more than $18 million lobbying outlay in 2018 as the company 
fended off privacy legislation and other initiatives that might impede its freedom to capture and process 
behavioral surplus. Google was also among the wealthiest of all registered lobbyists in the EU, second 
only to a lobbying group that represents a confederation of European corporations.1@ 

The firm also learned to engineer sophisticated lobbying efforts at the state level, primarily geared to 


fight back any proposed legislation that would augment privacy and curtail behavioral surplus operations. 
For example, Google won the right to put its self-driving cars on the road—anticipated as important 
supply chains—after enlisting Obama officials to lobby state regulators for key legislation122 Both 
Google and Facebook currently lead aggressive state-level lobbying campaigns aimed at repelling or 
weakening statutes to regulate biometric data and protect privacy. As one report put it, “They want your 
body.” #2 

In the fourth arena of fortifications the corporation learned to infiltrate and influence academic 
research and civil society advocacy in ways that softened or in some cases thwarted the examination of its 
practices. The Washington Post describes Google as a “master of Washington influence” and notes the 
subtlety with which the firm grasps and directs its own narrative. Schmidt was hands-on in this work as 
well. Already a board member of the New America Foundation, a public-policy think tank that played an 
influential role in shaping the Obama administration’s approach to economic issues, he assumed the 
chairmanship in 2013 with a personal donation of $1 million, a significant percentage of its $12.9 million 
annual budget that year. Between 1999 and 2016, when Schmidt left the board, the foundation received 
$21 million from a combination of Google, Schmidt, and Schmidt’s family foundation. 

The Washington Post published an elaborate exposé of Google’s meticulous work in this fourth 
domain, illustrated in the backstage intrigues that accompanied a three-part series on internet search 
competition held at George Mason University’s Law and Economics Center, a “free-market—oriented” 
academic center that had received significant funding from Google.44 The meetings occurred in May 
2012, just as the FTC was investigating the Google antitrust case. Reporters found that company staffers 
worked closely with the center, choosing pro-Google speakers and participants, many of whom were 
Google employees. Their efforts included “sending the center’s staff a detailed spreadsheet listing 
members of Congress, FTC commissioners, and senior officials with the Justice Department and state 
attorney general’s office.” Reporters noted that the conference’s panels were dominated by “leading 
technology and legal experts” who forcefully rejected the need for government action against Google, 
“making their arguments before some of the very regulators who would help determine its fate.” Many 
participants had no idea that Google was involved in crafting the meetings because Google and center 
staffers had agreed to conceal the corporation’s backstage involvement. 

The FTC antitrust investigation appears to have heightened Google’s fears of a regulatory threat to 
surveillance capitalism. That year, Google’s grant-making operation aimed at civil society organizations 
took an aggressive turn. According to the Center for Media and Democracy’s investigatory research 
report, “The Googlization of the Far Right,” the corporation’s 2012 list of grantees featured a new group 
of antigovernment groups known for their opposition to regulation and taxes and their support for climate- 
change denial, including Grover Norquist’s Americans for Tax Reform, the Koch brothers—funded 
Heritage Action, and other antiregulatory groups such as the Federalist Society and the Cato Institute 4 
The corporation also quietly acknowledged its membership in the corporate lobbying group ALEC, 
known for its opposition to gun control and emissions curbs, and for its support for voter-suppression 
schemes, tobacco industry tax breaks, and other far-right causes.4/2 Meanwhile, a list of Google Policy 
Fellows for 2014 included individuals from a range of nonprofit organizations whom one would expect to 
be leading the fight against that corporation’s concentrations of information and power, including the 
Center for Democracy and Technology, the Electronic Frontier Foundation, the Future of Privacy Forum, 
the National Consumers League, the Citizen Lab, and the Asociaci6n por los Derechos Civiles.® 

In July 2017 the Wall Street Journal reported that since 2009, Google had actively sought out and 
provided funding to university professors for research and policy papers that support Google’s positions 
on matters related to law, regulation, competition, patents, and so forth.“ In many cases, Google weighed 


in on the papers before publication, and some of the authors did not disclose Google as a source of 
funding. Although Google publicly claimed that “the check came with no requirements,” another case in 
2017 belied that notion. That summer, one of the New America Foundation’s most highly regarded 
scholars and a specialist in digital monopolies, Barry Lynn, posted a statement praising the EU’s historic 
decision to levy a $2.7 billion fine on Google as the result of a multiyear antitrust investigation. 
According to the New York Times and Lynn’s own account, New America’s director bent to pressure from 
Schmidt, firing Lynn and his Open Markets team of ten researchers. “Google is very aggressive in 
throwing its money around Washington and Brussels, and then pulling strings,” Lynn told the New York 
Times. “People are so afraid of Google now.” The reporters cite Google’s “muscular and sophisticated” 
influence operation as surpassing any other US company’s.12 

With Google in the lead, surveillance capitalism vastly expanded the market dynamic as it learned to 
expropriate human experience and translate it into coveted behavioral predictions. Google and this larger 
surveillance project have been birthed, sheltered, and nurtured to success by the historical conditions of 
their era—second-modernity needs, the neoliberal inheritance, and the realpolitik of surveillance 
exceptionalism—as well as by their own purpose-built fortifications designed to protect supply chain 
operations from scrutiny through political and cultural capture. 

Surveillance capitalism’s ability to keep democracy at bay produced these stark facts. Two men at 
Google who do not enjoy the legitimacy of the vote, democratic oversight, or the demands of shareholder 
governance exercise control over the organization and presentation of the world’s information. One man 
at Facebook who does not enjoy the legitimacy of the vote, democratic oversight, or the demands of 
shareholder governance exercises control over an increasingly universal means of social connection 
along with the information concealed in its networks. 


CHAPTER FIVE 


THE ELABORATION OF SURVEILLANCE 
CAPITALISM: KIDNAP, CORNER, COMPETE 


All words like Peace and Love, 
all sane affirmative speech, 
had been soiled, profaned, debased 
to a horrid mechanical screech. 


—W. H. AUDEN 
“WE TOO HAD KNOWN GOLDEN HOURS” 


I, The Extraction Imperative 


“Our ultimate ambition is to transform the overall Google experience, making it beautifully simple,” Larry 
Page said, “almost automagical because we understand what you want and can deliver it instantly.”! In 
the drive to fulfill this ambition, the extraction imperative produces a relentless push for scale in supply 
operations. There can be no boundaries that limit scale in the hunt for behavioral surplus, no territory 
exempted from plunder. The assertion of decision rights over the expropriation of human experience, its 
translation into data, and the uses of those data are collateral to this process, inseparable as a shadow. 
This explains why Google’s supply chains began with Search but steadily expanded to encompass new 
and even-more-ambitious territories far from clicks and queries. Google’s stores of behavioral surplus 
now embrace everything in the online milieu: searches, e-mails, texts, photos, songs, messages, videos, 
locations, communication patterns, attitudes, preferences, interests, faces, emotions, illnesses, social 
networks, purchases, and so on. A new continent of behavioral surplus is spun each moment from the 
many virtual threads of our everyday lives as they collide with Google, Facebook, and, more generally, 
every aspect of the internet’s computer-mediated architecture. Indeed, under the direction of surveillance 
capitalism the global reach of computer mediation is repurposed as an extraction architecture. 

This process originated online but has spread to the real world as well, a fact that we will examine 
more closely in Part II. If Google is a search company, why is it investing in smart-home devices, 
wearables, and self-driving cars? If Facebook is a social network, why is it developing drones and 
augmented reality? This diversity sometimes confounds observers but is generally applauded as visionary 
investment: far-out bets on the future. In fact, activities that appear to be varied and even scattershot 
across a random selection of industries and projects are actually all the same activity guided by the same 


aim: behavioral surplus capture. Each is a slightly different configuration of hardware, software, 
algorithms, sensors, and comnectivity designed to mimic a car, shirt, cell phone, book, video, robot, chip, 
drone, camera, cornea, tree, television, watch, nanobot, intestinal flora, or any online service, but they all 
share the same purpose: behavioral surplus capture. 

Google is a shape-shifter, but each shape harbors the same aim: to hunt and capture raw material. 
Baby, won’t you ride my car? Talk to my phone? Wear my shirt? Use my map? In all these cases the 
varied torrent of creative shapes is the sideshow to the main event: the continuous expansion of the 
extraction architecture to acquire raw material at scale to feed an expensive production process that 
makes prediction products that attract and retain more customers. When confronted in 2008 with a 
question about why Google had 150 “products,” its CEO, Eric Schmidt, responded: “That can be stated as 
criticism, but it can also be stated as strategy. The goal of the company is customer satisfaction. You 
should think of Google as one product: customer satisfaction.”? Those customers are the world’s 
advertisers and others who pay for its predictions. “Customer satisfaction” therefore equates to Google’s 
dominant market share in lucrative new behavioral futures markets, fed by its ever-expanding extraction 
architecture. 

New supply routes are continuously constructed and tested, and only some go operational. Routes that 
reliably produce scale, such as the Android smartphone operating system or Gmail, are elaborated and 
institutionalized. Those that fail are shuttered or modified. If one route is blocked, another is found. 
Successful supply routes double as canvases for targeted advertising, expanding the reach of behavioral 
futures markets and simultaneously engaging users in ways that yield yet more behavioral surplus. There 
will always be a changing roster of supply routes, but all variations share the same operational mandate: 
the capture of behavioral surplus and the acquisition of decision rights. Like a river running to the sea, if 
one route is blocked, another is found. 

In this chapter we follow the consequences of the extraction imperative as it drives the elaboration of 
the new market form and its competitive dynamics. The imperative elevates surplus supply operations to a 
defining role in every aspect of the surveillance capitalist enterprise. This begins with a continuous 
parade of innovations aimed at cornering raw-material supplies. Cornering is not simply a technological 
achievement. Sustainable dispossession requires a highly orchestrated and carefully phased amalgam of 
political, communicative, administrative, legal, and material strategies that audaciously asserts and 
tirelessly defends claims to new ground. The success of these strategies, first at Google and later at 
Facebook, established both their feasibility and their rewards, drawing new competitors into an 
increasingly ruthless cycle of kidnapping human experience, cornering surplus supplies, and competing in 
new behavioral futures markets. 


IT. Cornered 


The discovery of behavioral surplus in 2001-2002 meant that Google Search would be the first Google 
“service” to be re-crafted as a supply route. The resulting shifts in the mechanisms of Search were nearly 
impossible for people to imagine, let alone detect. When Harvard Business School’s Benjamin Edelman 
researched these hidden mechanisms in 2010, he found that the “enhanced features” option of a product 
called Google Toolbar—a plug-in for Microsoft’s Internet Explorer web browser that lets users search 
without having to go to google.com—transmits to the company “the full URL of every page view, 
including searches at competing search engines.” Edelman discovered that it was “strikingly easy” to 
activate this option but impossible to disable it. Even when a user specifically instructed that the toolbar 


be disabled, and even when it appeared to be disabled because it had disappeared from view, the toolbar 
continued to track browsing behavior.? As Google now hosts “trillions” of searches annually, its varied 
search-related tracking mechanisms combined with its robust and nearly inescapable cookies (bits of 
tracking code inserted in your computer) ensure immense economies of scale that constitute the bedrock of 
Google’s supply operations.4 
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Surveillance capitalism’s master motion is the 
accumulation of new sources of behavioral 
surplus with more predictive power. The goal 
is predictions comparable to guaranteed 
outcomes in real-life behavior. Extraction 
begins online, but the prediction imperative 
increases the momentum, driving extraction 
toward new sources in the real world. 


Shoshana Zuboff, 
The Age of Surveillance Capitalism 


Figure 3: The Dynamic of Behavioral Surplus Accumulation 


In 2015 internet legal scholar Tim Wu joined Harvard Business School’s Michael Luca and a team of 
data scientists from Yelp to research hidden mechanisms in Google Search that function, from our 
perspective, to expand crucial supply functions. They discovered that Google was systematically 
corrupting Search results to favor its own content and “downstream products”: 


Google has begun to develop its own content over time, such as its own price results for shopping and its own reviews for local 
businesses.... Google is acting both as a search engine and a content provider. To use its search dominance to promote this content, 


Google has developed a feature called “universal search,” through which it intentionally excludes content competitors and only shows 


Google’s content.2 


The ceaseless requirement for surplus at scale predicts corporate behavior that favors exclusivity. 
Because Search is the foundation of Google’s supply operations, the company has every incentive to 
entice users to its Search platform, content, and ancillary services and then to use its backstage “methods, 
apparatus, and data structures” for efficient extraction. The tilt toward exclusivity produces a range of 
practices considered “monopolistic” in the perspective of twentieth-century regulatory frameworks. 
These characterizations, while valid, omit the most-salient elements of the new order. The extraction 
imperative demands that everything be possessed. In this new context, goods and services are merely 
surveillance-bound supply routes. It’s not the car; it’s the behavioral data from driving the car. It’s not the 
map; it’s the behavioral data from interacting with the map. The ideal here is continuously expanding 
borders that eventually describe the world and everything in it, all the time. 

Traditionally, monopolies on goods and services disfigure markets by unfairly eliminating competition 
in order to raise prices at will. Under surveillance capitalism, however, many of the practices defined as 
monopolistic actually function as means of cornering user-derived raw-material supplies. There is no 
monetary price for the user to pay, only an opportunity for the company to extract data. Cornering 
practices are not designed to protect product niches but rather to protect critical supply routes for the 
unregulated commodity that is behavioral surplus. In another time, rogue market actors might corner 
markets in copper or magnesium, but in our time it is behavioral surplus. The corporation unfairly 
impedes competitors in Search in order to protect the dominance of its most important supply route, not 
primarily to fix prices. 

These cornering operations are not abstractions, with distant effects on minerals or crops that 
eventually wind their way toward the price of goods. In this picture it is we who are “cornered.” We are 
the source of the coveted commodity; our experience is the target of extraction. As surveillance capitalism 
migrates from Silicon Valley to a range of other firms and sectors, we gradually find ourselves in a world 
of no escape, “cornered” by converging, overlapping, and relentlessly expanding dispossession 
operations. It is important to say—and we will revisit this theme more than once—that regulatory 
interventions designed to constrain Google’s monopoly practices are likely to have little effect on the 
fundamental operations of this market form. New supply routes are continuously discovered, opened, and 
secured. Dispossession activities are compelled to circumvent every obstacle and will continue to do so, 
short of a genuine existential threat. 

Google’s Android mobile platform offers an example of the governing role of surplus capture and 
defense. Internet use went mobile with the rise of the smartphone and the tablet, and Google was forced to 
find new ways to defend and expand its primary supply chain in Search. Android quickly became the 
corporation’s second critical supply route for behavioral surplus. In 2008 Google led an alliance of 
technology manufacturers and wireless operators to develop an “open and comprehensive platform for 
mobile devices.” Some observers thought that an Android phone was Google’s opportunity to compete 


with Apple for the lucrative margins on smartphones, but Google insiders had grasped the even greater 
potential for growth and profit through behavioral surplus and its fabrication into prediction products. 

Google licensed Android to mobile handset makers for free because it was intended to draw users into 
Google Search and other Google services, establishing a ubiquitous mobile supply apparatus to sustain 
known terrains of behavioral surplus and open up new ones, including geolocation and mobile payment 
systems that are highly coveted by advertisers.° As Google’s chief financial officer told financial analysts 
in 2009, “If we move forward the adoption of these mobile phones by lowering the cost because it is open 
source, think of how many searches [that will produce].”” A prominent Silicon Valley venture capitalist 
described Android devices in 2011 as 


not “products” in the classic business sense... they are not trying to make a profit on Android.... They want to take any layer that 


lives between themselves and the consumer and make it free (or even less than free).... In essence, they are not just building a moat; 


Google is also scorching the earth for 250 miles around the outside of the castle to ensure no one can approach it8 


Supply operations were the protected treasure within the fortified castle, and Android’s development 
policies were key to the success of this supply strategy. Unlike the iPhone, the Android platform was 
“open source,” which made it easy for applications developers around the world to create apps for 
Android users. Eventually, Google bundled this valuable new universe of apps into its Google Play store. 
Manufacturers who wanted to preinstall Google Play on their devices were required to license and install 
Google’s mobile services as exclusive or default capabilities: Search, Gmail, Google Pay, YouTube, 
Google Maps, Google Photos, and whatever other supply routes happen to be in ascendance at the time. 

In 2016 Google’s Android practices became the focus of a European Union antitrust investigation, and 
its complaints were a mirror image of Google’s purposeful construction and protection of vital supply 
routes in Search and Mobile. Once again, governmental opposition to Google’s monopolistic activities 
emphasized traditional competitive harms at the expense of surveillance capitalism’s new harms. By 
April of 2013, Eric Schmidt told a conference devoted to “all things” digital that “our goal with Android 
is to reach everyone. We’ll cross one billion Android devices in six to nine months. In a year or two, 
we’ll hit two billion.... A relatively inexpensive smartphone with a browser is all you need to get the 
world’s information.” That final sentence presumably was intended to describe benefits to Android’s 
users. However, it is an even-more-effective summary of Google’s own ambitions and an insight into the 
vital economies of scale associated with this mobile supply route.? 

Google fiercely defends threatened supply routes. Any disruption of its extraction operations and its 
exclusive claims to raw material is the line that cannot be breached. In 2009 Android manufacturer 
Motorola chose to replace Google’s free location services with those of Skyhook Wireless, which, 
Motorola believed, produced more-reliable results. A Google product manager admitted Skyhook’s 
superiority but expressed his concerns in an e-mail to a Google executive, noting that if other 
manufacturers switched to Skyhook, it “would be awful for Google, because it will cut off our ability to 
continue collecting data” for the company’s Wi-Fi location database. Court documents from Skyhook’s 
eventual lawsuit against Motorola (and Samsung) include an e-mail from Google’s senior vice president 
of Mobile to Motorola’s CEO, insisting that the interruption of Google’s data collection was a “stop-ship 
issue.” 

Another legal fracas further illustrates how products such as Android are valued more for supply than 
for sales. Disconnect, Inc., founded in 2011 by two former Google engineers and a privacy-rights 
attorney, developed desktop and mobile applications “to protect the privacy and security of internet users 
by blocking invisible, unsolicited network connections between a user’s browser or mobile device and 


sites/services that engage in invisible tracking or are known or suspected distributors of malware... not 
only when he browses the web, but also when he uses other third-party mobile applications.” 
Disconnect specifically took aim at the “invisible, unsolicited and frequently undisclosed” network 
connections from third-party sites and services that occur as soon as you visit a website or open a mobile 
application. 

Unfortunately for Disconnect, the very process that it aimed to impede had been established as a 
significant supply route for Google and other surveillance capitalists.4 Several studies explain the extent 
of Google’s extraction architecture, including the Web Privacy Census, which primarily measured 
cookies. The census analyzed the top 100, 1,000, and 25,000 websites in 2011, 2012, and 2015, years of 
feverish discovery and elaboration for surveillance capitalists. The comparisons between 2012 and 2015 
revealed more than twice as many sites with 100 or more cookies and more than three times the number of 
sites with 150 or more cookies. In 2015 the team found that anyone who simply visited the 100 most 
popular websites would collect over 6,000 cookies in his or her computer, 83 percent of which were 
from third parties unrelated to the website that was visited. The census found “Google tracking 
infrastructure” on 92 of the top 100 sites and 923 of the top 1,000 sites, concluding that “Google’s ability 
to track users on popular websites is unparalleled, and it approaches the level of surveillance that only an 
Internet Service Provider can achieve.”!2 

Another 2015 analysis, this one of the top one million websites, by Timothy Libert of the University of 
Pennsylvania, found that 90 percent leak data to an average of nine external domains that track, capture, 
and expropriate user data for commercial purposes. Among these websites, 78 percent initiate third-party 
transfers to a domain owned by one company: Google. Another 34 percent transfer to a Facebook-owned 
domain.4 Steven Englehardt and Arvind Narayanan from Princeton University reported in 2016 on the 
results of their measurement and analysis of tracking data from one million websites. They identified 
81,000 third parties, but only 123 of those are present on more than 1 percent of sites. Of that group, the 
top five third parties and twelve of the top twenty are Google-owned domains. “In fact,” they conclude, 
“Google, Facebook, and Twitter are the only third-party entities present on more than 10% of sites.” 
Chinese researchers investigated 10,000 apps from the top third-party app markets in 2017. They found a 
“covert” process in which an app autonomously launches other apps in the background of your phone, and 
they concluded that this “app collusion” was most prevalent in third-party Android markets. Of the 1,000 
top apps on one of China’s popular platforms, 822 launched an average of 76 other apps, and of these 
launches, 77 percent were triggered by cloud-based “push services” that are intended to update apps but 
are obviously doing a lot more. In the Android environment, the researchers note, Google provides the 
push service.1® 

Finally, extraordinary research from the French nonprofit Exodus Privacy and the Yale Privacy Lab in 
2017 documented the exponential proliferation of tracking software. Exodus identified 44 trackers in 
more than 300 apps for Google’s Android platform, some of which are also produced for Apple’s 
operating system. Altogether, these apps have been downloaded billions of times. Two themes stand out 
in the research report: ubiquity and intensification. First, there is hardly an innocent app; if it’s not 
tracking you now, it may be doing so in the next week or month: “There is an entire industry based upon 
these trackers, and apps identified as ‘clean’ today may contain trackers that have not yet been identified. 
Tracker code may also be added by developers to new versions of apps in the future.” Second is that even 
the most innocent-seeming applications such as weather, flashlights, ride sharing, and dating apps are 
“infested” with dozens of tracking programs that rely on increasingly bizarre, aggressive, and illegible 
tactics to collect massive amounts of behavioral surplus ultimately directed at ad targeting. For example, 
the ad tracker FidZup developed “communication between a sonic emitter and a mobile phone....” It can 


detect the presence of mobile phones and therefore their owners by diffusing a tone, inaudible to the 
human ear, inside a building: “Users installing ‘Bottin Gourmand,’ a guide to restaurants and hotels in 
France, would thus have their physical location tracked via retail outlet speakers as they move around 
Paris. Their experience would be shared by readers of a car magazine app ‘Auto Journal’ and the TV 
guide app ‘TeleStar.’” In a pattern foreshadowed by the Google patent that we examined in Chapter 3 and 
that we shall see repeatedly in the coming chapters, the research findings emphasize that the always-on 
tracking is impervious to the Android “permissions system,” despite its promises of user control.“ 

Given the hostility and intensity of these supply operations, it is not too surprising that Disconnect 
software was banned from Google Play’s vast catalog of mobile apps, leading to Disconnect’s lawsuit 
against Google in 2015. The startup’s complaint explains that “advertising companies including Google 
use these invisible connections to ‘track’ the user as he/she browses the web or opens other mobile 
applications, in order to collect personal information about the user, create a ‘profile’ of the user, and 
make money targeting advertising to the user.”!8 It went on to argue that the privacy protections offered by 
Google “invariably permit the company to continue to gather private information....”!2 Google’s ban of 
the Disconnect app is exceptionally revealing in light of the fact that, unlike Apple, Google is notoriously 
“libertarian” when it comes to the millions of apps sold or downloaded for “free” from its app store. Its 
loose guidelines attempt to identify and exclude malicious applications but little else.22 

Disconnect’s founders attempted to challenge the extraction imperative, but they could not do so alone. 
After trying—without success—to negotiate with Google, they eventually joined other organizations in 
filing a complaint against Google in the EU, helping to precipitate an Android-focused antitrust probe.#! 
As Disconnect argued, 


Google is under enormous pressure from the financial community to increase the “effectiveness” of its tracking, so that it can 


increase revenues and profits. Giving a user the ability to control his own privacy information (and to protect himself from malware) 


by blocking invisible connections to problematic sites constitutes an existential threat to Google.22 


As ex-Googlers, the founders of Disconnect thought that they knew their adversary well, but they 
underestimated the progress of surveillance capitalism’s institutionalization and the ferocity with which 
the corporation was prepared to fend off “existential threats” to its supply routes. 


III. The Dispossession Cycle 


Long before Disconnect, Google had discovered that successful dispossession is not a single action but 
rather an intricate convergence of political, social, administrative, and technical operations that requires 
cunning management over a substantial period of time. Its dispossession operations reveal a predictable 
sequence of stages that must be crafted and orchestrated in great detail in order to achieve their ultimate 
destination as a system of facts through which surplus extraction is normalized. 

The four stages of the cycle are incursion, habituation, adaptation, and redirection. Taken together, 
these stages constitute a “theory of change” that describes and predicts dispossession as a political and 
cultural operation supported by an elaborate range of administrative, technical, and material capabilities. 
There are many vivid examples of this cycle, including Google’s Gmail; Google’s efforts to establish 
supply routes in social networks, first with Buzz and then with Google+; and the company’s development 
of Google Glass. In this chapter we focus on the Street View narrative for a close look at the 
dispossession cycle and its management challenges. 


The first stage of successful dispossession is initiated by unilateral incursion into undefended space: 
your laptop, your phone, a web page, the street where you live, an e-mail to your friend, your walk in the 
park, browsing online for a birthday gift, sharing photos of your kids, your interests and tastes, your 
digestion, your tears, your attention, your feelings, your face. The incursion is when dispossession 
operations rely on their virtual capabilities to kidnap behavioral surplus from the nonmarket spaces of 
everyday life where it lives. The incursion initiates Google’s most basic and prolific form of 
dispossession: Arendt’s repeated “original sin of simple robbery.” Incursion moves down the road 
without looking left or right, continuously laying claim to decision rights over whatever is in its path. “I’m 
taking this,” it says. “These are mine now.” 

The company has learned to launch incursions and proceed until resistance is encountered. It then 
seduces, ignores, overwhelms, or simply exhausts its adversaries. Seduction means a cascade of golden 
enticements: unprecedented storage, access to new qualities of information, new conveniences. When 
necessary, the company can just as easily pivot toward harsher tactics that aim to deplete its adversaries’ 
time, money, and grit. There are hundreds of cases launched against Google by countries, states, groups, 
and individuals, and there are many more cases that never become public. According to Marc Rotenberg, 
executive director of the Electronic Privacy Information Center (EPIC), no one knows precisely how 
many lawsuits there are around the world.” The legal challenges are varied, but they nearly always come 
back to the same thing: unilateral incursion met by resistance. 

Legal opposition and social protest have surfaced in relation to the digitalization of books,~ the 
collection of personal information through Street View’s Wi-Fi and camera capabilities,” the capture of 
voice communications,2° the bypassing of privacy settings,2’ the manipulation of search results,“? the 
extensive retention of search data,2” the tracking of smartphone location data,22 wearable technologies and 
facial-recognition capabilities,*! the secret collection of student data for commercial purposes, * and the 
consolidation of user profiles across all Google’s services and devices,° just to name several instances. 
Expect to see drones, body sensors, neurotransmitters, “digital assistants,” and other sensored devices on 
this list in the years to come. Meanwhile, Google is consistently stunning in its sense of entitlement, 
resolve, and audacity. The extraction imperative compels it to push new boundaries into undefended 
space. 

In a second stage the aim is habituation. Whereas lawsuits and investigations unwind at the tedious 
pace of democratic institutions, Google continues the development of its contested practices at high 
velocity. During the elapsed time of FTC and FCC inquiries, court cases, judicial reviews, and EU 
Commission investigations, the new contested practices become more firmly established as institutional 
facts, rapidly bolstered by growing ecosystems of stakeholders. People habituate to the incursion with 
some combination of agreement, helplessness, and resignation. The sense of astonishment and outrage 
dissipates. The incursion itself, once unthinkable, slowly worms its way into the ordinary. Worse still, it 
gradually comes to seem inevitable. New dependencies develop. As populations grow numb, it becomes 
more difficult for individuals and groups to complain. 

In a third stage of the cycle, when Google is occasionally forced to alter its practices, its executives 
and engineers produce superficial but tactically effective adaptations that satisfy the immediate demands 
of government authorities, court rulings, and public opinion. Meanwhile, in a final stage the corporation 
regroups to cultivate new rhetoric, methods, and design elements that redirect contested supply operations 
just enough so that they appear to be compliant with social and legal demands. The creativity, financial 
resources, and determination brought to the task of managing this staged process are flexible and dynamic. 
In contrast, the operational necessity for economies of scale in the capture of behavioral surplus is a 
perpetual-motion machine whose implacable rhythms offer no room for divergence. 


The theory and practice of dispossession were developed and refined as the company learned how to 
counter and transform public resistance as an essential condition for the protection and expansion of its 
behavioral surplus franchise. Google’s launch of Gmail on April Fool’s Day, 2004, provided an early 
occasion to climb this learning curve as the corporation faced down public outrage over the automated 
scanning of e-mail content intended as a fresh source of surplus for targeted ads. Eventually, the 
dispossession cycle was refined as an explicit theory of change that framed a tactical game plan, which is 
by now regularly evoked as the surveillance capitalist corporation’s battle-tested response to societal 
resistance. 

The dispossession cycle at Google was so successful in facing down the threats to Gmail that it was 
replicated and further elaborated in the battles over Google Street View, the street-mapping operation 
launched in 2007. Once again, the company did not ask permission. It simply repeated the “original sin of 
simple robbery” and took what it wanted, waiting for resistance to run its course as it devoured and 
datafied the world’s public spaces, streets, buildings, and homes. 


Stage One: Incursion 


Street View first entered public awareness with an apparently benign blog post. Peter Fleischer, Google’s 
“privacy counsel,” helped launch the new “service” by writing a paean celebrating America’s “noble 
tradition” of public spaces, where, he claimed, “people don’t have the same expectations of privacy as 
they do in their homes.” As a lawyer, Fleischer knows the work that words do in establishing contracts 
and setting precedents, so it pays to read his 2007 words with care. His casually declarative prose 
undertakes some extraordinary work as he asserts that all public spaces are fair game for Google’s taking. 
In his account, any public space is a fitting subject for the firm’s new breed of incursion without 
authorization, knowledge, or agreement. Homes, streets, neighborhoods, villages, towns, cities: they are 
no longer local scenes where neighbors live and walk, where residents meet and talk. Google Street 
View, we are informed, claims every place as just another object among objects in an infinite grid of GPS 
coordinates and camera angles. 

In his declaration, Fleischer intends to establish Google’s prerogative to empty every place of the 
subjective meanings that unite the human beings who gather there. Yes, when we leave our home we know 
we will be seen, but we expect to be seen by one another in spaces that we choose. Instead, it’s all 
impersonal spectacle now. My house, my street, my neighborhood, my favorite café: each is redefined as 
a living tourist brochure, surveillance target, and strip mine, an object for universal inspection and 
commercial expropriation. 

Google had already taken everything on the web, but Street View and Google’s other mapping 
operations, Google Maps and Google Earth (the company’s 3-D view of the world using satellite and 
aerial imagery), announced an even-more-ambitious vision. Everything in the world was to be known and 
rendered by Google, accessed through Google, and indexed by Google in its infinite appetite for 
behavioral surplus. The presumption is that nothing is beyond Google’s borders. The world is vanquished 
now, on its knees, and brought to you by Google. 

The blog post that accompanied Street View is a precise replica of the invaders who once landed on 
that blameless Caribbean beach. Those adelantados concealed the bare facts of invasion in elaborate 
gestures of friendship and humility that made it impossible to discern the clear and present danger implicit 
in their arrival. Fleischer similarly assures his audience of friendly terms. Street View, which used 
cartoonishly wrapped cars with a large 360-degree camera mount on the roof to capture the imagery it 
sought, was designed to “respect the privacy of people who happen to be walking down a public street,” 


Fleischer wrote. “That’s why we designed a simple process for anyone to contact us and have their image 
removed,” and he promised that it would respect laws and customs “in other parts of the world.”=4 

Resistance came swiftly and often. By January 2009, Street View was facing opposition in Germany 
and Japan. John Hanke, by then the vice president for Google Maps-related products, dismissed the 
uproar. (You will recall that Hanke had founded the CIA-funded satellite mapping company, Keyhole, and 
after Google’s purchase he led its transformation into Google Earth.) He told a reporter that it was all 
simply part of a “cycle of people understanding exactly what it is and what it isn’t and what they shouldn’t 
really be concerned about”—in other words, the dispossession cycle. Google Earth was also under fire, 
blamed for aiding a deadly terrorist attack in Mumbai, but Hanke insisted that the debate over Google 
Earth or Street View had “mostly died off” in “the West.” He cleverly equated any resistance to Google’s 
incursions with the anti-freedom-of-expression interests of authoritarian governments and their “closed 
information societies.”22 This would become a standard rhetorical device for Google and its allies as 
they executed their offense. 

Was Hanke surprised, then, when in April 2009 residents of the quiet English village of Broughton 
blocked a Street View car that tried to breach the village perimeter, calling it an unwelcome intrusion? 
This was “the West,” after all, but the debate over privacy, self-determination, and decision rights was 
anything but dead. Privacy International submitted a formal complaint to the UK privacy authority, citing 
more than 200 reports from people who were identifiable on Street View images and demanded that the 
service be suspended. 

Google’s executives had apparently missed Fleischer’s memo on respecting privacy claims. Instead, 
Hanke dismissed the protesters out of hand. He told the London Times that the company was undeterred 
and planned to complete coverage of the UK by the end of that year. He declared that Street View’s 
information was “good for the economy and good for us as individuals.... It is about giving people 
powerful information so that they can make better choices.”2° 

Hanke’s remarks were wishful thinking, of course, but they were consistent with Google’s wider 
practice: it’s great to empower people, but not too much, lest they notice the pilfering of their decision 
rights and try to reclaim them. The firm wants to enable people to make better choices, but not if those 
choices impede Google’s own imperatives. Google’s ideal society is a population of distant users, not a 
citizenry. It idealizes people who are informed, but only in the ways that the corporation chooses. It means 
for us to be docile, harmonious, and, above all, grateful. 

In 2010 the German Federal Commission for Data Protection announced that Google’s Street View 
operation actually camouflaged a covert data sweep; Street View cars were secretly collecting personal 
data from private Wi-Fi networks.*2 Google denied the charge, insisting that it was gathering only 
publicly broadcast Wi-Fi network names and the identifying addresses of Wi-Fi routers, but not personal 
information sent over the network.*® Within days, an independent analysis by German security experts 
proved decisively that Street View’s cars were extracting unencrypted personal information from homes. 
Google was forced to concede that it had intercepted and stored “payload data,” personal information 
grabbed from unencrypted Wi-Fi transmissions. As its apologetic blog post noted, “In some instances 
entire emails and URLs were captured, as well as passwords.” Technical experts in Canada, France, and 
the Netherlands discovered that the payload data included names, telephone numbers, credit information, 
passwords, messages, e-mails, and chat transcripts, as well as records of online dating, pornography, 
browsing behavior, medical information, location data, photos, and video and audio files. They concluded 
that such data packets could be stitched together for a detailed profile of an identifiable person.” 

Google’s “Spy-Fi” scandal filled headlines around the world. Many believed that the Street View 
revelations would inflict irreparable damage to Google. In Germany, where the firm’s actions were in 


clear violation of privacy and data-protection laws, officials reacted angrily and warned that Google 
would face EU investigations and consequences in the German courts. A bill was introduced into the 
German Parliament that proposed to fine Google for displaying personal property without owners’ 
consent. Google faced fresh litigation in Switzerland, Canada, France, and the Netherlands. By 2012, 
there were multiple investigations in twelve countries, including most of Europe, the North Atlantic, and 
Australia, and Google had been found guilty of violating laws in at least nine countries.” 

In the US, attorneys general from thirty-eight states launched a probe into Google’s Street View 
practices. Private citizens filed numerous class-action suits, eight of which were consolidated in the 
Northern California US District Court. The head of Privacy International said that Google was becoming 
“Big Brother.”*! The Electronic Privacy Information Center championed substantial legal resistance in the 
US against Google’s efforts to avoid repercussions in the wake of the Spy-Fi scandal, and it maintained a 
detailed and continuously updated online overview of the worldwide outrage, protests, investigations, 
litigation, and settlements in response to Google Street View and its extraction tactics.“ 

Google characterized Street View’s “privacy violations” as a “mistake” made by a single engineer 
working on an “experimental” project, whose code had inadvertently made it into Street View’s software. 
The firm refused to release the identity of the mystery engineer and insisted that the project’s leaders were 
unaware of the data capture and “had no intention” of using those data. As Eric Schmidt told the 
Financial Times, “We screwed up,” noting that the engineer in question would face an internal 
investigation for his clear “violation” of Google’s policies. Unbowed, Schmidt insisted on the validity of 
Google’s mission to index all the world’s information.” 

A 2012 investigation by the Federal Communications Commission described the case as “a deliberate 
software-design decision by one of the Google employees working on the Street View project.” The 
engineer had been selected for the team because of his unique expertise in Wi-Fi “wardriving,” the 
practice of driving around using equipment to locate wireless networks.” His design notes indicated that 
user traffic and location data would be logged along with “information about what they are doing” that 
would “be analyzed offline for use in other initiatives.” The notes identified but then dismissed “privacy 
considerations.”*® 

The FCC found evidence that contradicted Google’s scapegoating narrative. The records showed that 
the engineer had e-mailed links to his software documentation to project leaders, who then shared them 
with the entire Street View team. It also found evidence that on at least two occasions, the engineer told 
his colleagues that Street View was collecting personal data. Despite these facts along with evidence of 
the company’s exhaustive internal software reviews and testing procedures and the regular transfer of 
payload data from Street View’s hard disks to Google’s Oregon data center, Google’s engineers denied 
any knowledge of personal data collection.“ 


Stage Two: Habituation 


Hanke’s bet that the “cycle” would eventually wear down resistance reflects a key operational component 
of the extraction imperative, discovered in Search, refined with Gmail, and elaborated with Street View. 
The messages that come through are “Don’t look back. Wait them out. Step on them, if necessary.” 

The April 2012 FCC report is heart wrenching in its way, a melancholic depiction of democracy’s 
vulnerability in the face-off with a wealthy, determined, and audacious surveillance capitalist opponent. 
In November 2010 the FCC sent Google a letter of inquiry requesting necessary information. Little was 
forthcoming. By March of the next year, a second “supplemental” letter was sent. Google’s response was 
incomplete information and lack of cooperation, which produced another “demand letter” in August. 


Google’s continued lack of engagement required yet another letter in late October. The FCC staff was 
burdened with following up and chasing down evasive corporate executives and their representatives for 
an entire year. 

The document is a revelation of negative space and a saga of democracy rebuffed. The FCC’s detailed 
initial request produced “only five documents” and no e-mails. The corporation said that it had no time to 
undertake a comprehensive review, calling it “burdensome.” Google “failed” to identify relevant 
individuals. It “redacted” names. It asserted that the information requested “serves no useful purpose.” It 
“failed” to verify information. When asked for specific submissions, “Google did not do so.” Google 
“argued” that it should “not be required” to provide access to the payload data it had illicitly collected. 
“Google waited....” The phrases “failed to respond” and “failed to provide” are repeated throughout the 
account. “Google violated Commission orders... delaying....” Affidavits were requested five times, but 
the company did not provide any of these until September 2011, after the FCC threatened a subpoena. The 
mystery engineer simply refused to speak with investigators, citing his Fifth Amendment right to avoid 
self-incrimination. As the report concludes, “There is evidence that Google’s failure to cooperate with the 
Bureau was in many or all cases deliberate.” It might have said “imperative.” 

Ultimately, the corporation’s lawyers prevailed, defending Google’s data sweeps with a single 
obscure passage in a decades-old wiretap law. Perhaps the most telling element of the entire episode is 
that the same democratic system of laws and rules that the corporation openly treated with scorn was 
invoked to protect it from accountability. In the end the FCC fined Google only $25,000 for obstructing its 
investigation. Google did not evade legal consequences because society agreed with its practices, but 
because there was not enough relevant law to protect citizens from its incursions. 

The thirty-eight attorneys general did not fare much better. When the group’s leader, Connecticut’s 
Richard Blumenthal, issued a civil investigative demand (equivalent to a subpoena) to get access to the 
infamous private data, “Google ignored it.”“2 The company finally agreed to settle with the states in 2013, 
accepting a mere $7 million fine and a series of agreements regarding “aggressive” self-policing. The 
New York Times announced that Google had finally admitted that “it had violated people’s privacy during 
its Street View mapping project, when it casually scooped up... personal information,” as if this scandal 
had been the only contested element in the whole affair. State officials crowed that “the industry giant... is 
committing to change its corporate culture to encourage sensitivity to issues of personal data privacy.” 
In light of the fact that the extraction imperative is what makes this giant a giant, one doesn’t know 
whether to laugh or cry at the confidence of the attorneys general in Google’s commitment to privacy self- 
regulation. 

There are two key elements here that illuminate habituation tactics. The first is the simple fact of the 
elapsed time between Street View’s initial incursion in 2007, the 2010 scandal, the 2012 conclusion of 
the FCC inquiry, and the 2013 conclusion of the states’ investigation. The German investigation also 
closed in late 2012, with little to show for its trouble. Other contests and lawsuits lumbered on. Despite 
all the sound and fury, Google continued to operate Street View during those years. Between 2008 and 
2010, 600 billion bytes of personal information were collected “illegitimately” around the world, 200 
billion of those in the US.°2 The corporation said that it had discontinued personal data collection. Had it? 
Can anyone say with certainty? Even if it had, the original incursion that was Street View continued 
unscathed. 

The second point is that in retrospect, one sees that the very idea of a single rogue engineer was 
designed and elaborated as a brilliant piece of misdirection, a classic scapegoating ploy. It directed 
attention away from the ambitious and controversial agenda of the extraction imperative toward a 
different narrative of a single infected cell excised from the flesh of an enormous but innocent organism. 


All that was left was to excise the infected flesh and let the organism declare itself cured of its privacy 
kleptomania. Then—a return to the streets, born again. 

Google achieved exactly what Hanke had predicted. Street View’s fundamental audacity, the 
unprecedented astonishing incursion that drew English villagers into the streets to block a Google camera 
car, enjoyed six more years of rooting in global consciousness. The corporation’s strategic discipline 
when it comes to stonewalling, snubbing, and exploiting democracy brought six more years of people 
using Street View data and six more years to build the tacit case for Google’s inevitability and our 
helplessness. There were six more years for this simple robbery of decision rights to shade into normalcy 
and even to be reckoned as “convenient,” “useful,” or “marvelous.” 


Stage Three: Adaptation 


In October 2010, just before the corporation received the FCC’s first letter of inquiry, Google’s senior 
vice president of Engineering and Research announced “stronger privacy controls” in an official Google 
blog post. “We’ve failed badly here,” he said. The Street View scandal was framed as an inadvertent 
error, a single blemish on a company that works hard “to earn your trust.” The post assured the public that 
the corporation was talking to external regulators “about possible improvements to our policies” and 
promised that it would make changes to ensure user privacy. Alma Whitten, a Google executive with 
credentials in computer security and privacy controls, was appointed as director of Privacy across 
engineering and product management. The blog also introduced a new internal training emphasis on 
“responsible collection, use and handling of user data.” Finally, the post pledged new internal controls to 
oversee how data are handled. “We are mortified by what happened,” it read, “but confident that these 
changes to our processes and structure will significantly improve our internal privacy and security 
practices for the benefit of all our users.”22 

Though pledging reform to the public, the corporation was simultaneously forced to adapt to 
governmental demands in a range of countries—including Australia, Belgium, Canada, France, Holland, 
Hong Kong, Ireland, Israel, Italy, New Zealand, Poland, Spain, South Korea, the UK, and the US—where 
Street View was subjected to litigation, fines, and/or regulation. In Japan, homeowners complained of 
Street View cameras that peered above privacy fencing to record private homes. Google agreed to 
government demands to lower its cameras, reshoot all images, and blur identifiable facial images and 
license plates. In Germany, Google allowed residents to request that their homes be blurred in any Street 
View images. Nearly 250,000 households made opt-out requests in 2009-2010, requiring Google to hire 
200 temporary programmers to meet the demand.?2 Google was fined 145,000 euros by the Hamburg data- 
protection supervisor who had first discovered the Street View illicit data gathering, just short of the 
150,000 euro fee he could have imposed.°2 It was the largest fine ever to have been levied by European 
regulators for privacy concerns. The discount reflected Google’s assurances that it would swiftly and 
thoroughly delete the payload data. In 2011 Google ended its Street View program in Germany, continuing 
to support but no longer update the images it had already collected.4 

Other countries imposed bans on Street View operations. Switzerland initially banned the service in 
2009, insisting that Google remove all imagery it had posted of Swiss towns and cities. Eventually, the 
ban was lifted, but the Swiss Federal Administrative Court imposed a series of strict guidelines, 
including blurring faces, instituting opt-out measures, and reducing camera heights. By 2016, Google’s 
service remained confined to outdoor tourist sites.22 The corporation also faced Street View bans in 
Austria, the Czech Republic, Greece, India, and Lithuania. By the summer of 2017, though, Street View 


data were available from at least some regions of each of these countries.2° 


Stage Four: Redirection 


What Google did not say in its mea culpa blog post, the one thing that it could not say, was that it would 
abandon its fundamental logic of accumulation: the principles of surveillance capitalism that had brought 
the behemoth into existence and sustained its growth. The message of Street View’s redirection campaign 
was that Google would not exempt anything from the grid. Everything must be corralled for conversion 
into raw material. Short of institutional suicide, there is little that Google can say or do to ensure “user 
privacy.” This helps to explain why, as one 2015 article celebrating the history of Google Maps observes, 
“Google Maps was attracting all sorts of privacy controversies... people were freaked out.... But that 
doesn’t mean Street View has been tamped down as a project. It’s now available in 65 of Google Maps’ 
200-some countries.”2/ 

Alma Whitten’s job was to repair Google’s privacy reputation, not to dismantle the extraction 
imperative and its relentless demand for economies of scale in the supply function. This is to say that her 
job was a logical impossibility. That she may have nevertheless taken it seriously is suggested by the fact 
that just two-and-a-half years after her appointment as privacy czar, she announced her retirement from 
Google in April 2013. Indeed, it is painful to watch Whitten testify about Google’s practices to an early- 
2013 congressional hearing. She was under questioning from Congress, and one sees the effort required 
as she hunts for the words to convey an answer without conveying the truth.° The time had come to 
regroup and redirect the global mapping project, not to end it. 

That nothing much had changed or would change was immediately suggested by the fate of Google’s 
mystery engineer in the two years that followed the scandal. Within days of the FCC report in April 2012, 
a former state investigator who had been assigned to the Street View inquiry identified Google’s “rogue” 
actor as Marius Milner, a celebrated hacker and wardriving specialist. It had been two years since the 
supposedly irreparable damage that he inflicted on Google and his “clear violation” of policy, yet he 
continued to be employed at the firm in its YouTube operations. Later that year, he would be one of six 
inventors on a team led by John Hanke to patent “A System and Method for Transporting Virtual Objects 
in a Parallel Reality Game.” 

The invention in which Milner participated was related to a virtual reality game called Ingress, also 
developed by Hanke and his team at Google. (Hanke would eventually establish his own shop, Niantic 
Labs, within Google’s new Alphabet holding company.) Ingress became a test bed for many of the 
foundational concepts that reappeared in another “game,” Pokémon Go, a prototype of a second phase of 
surveillance capitalist expansion that we examine closely in Part II. In this next phase, Google’s maps are 
a critical resource for the expansion of digital dispossession from the virtual world to the one that we call 
“real.” In light of those plans, Street View could not be allowed to die or even to be constrained. The 
corporation’s senior product manager for Google Maps framed it succinctly in September 2012, just four 
months after the FCC investigation: “If you look at the offline world, the real world in which we live, that 
information is not entirely online. Increasingly as we go about our lives, we are trying to bridge that gap 
between what we see in the real world and [the online world], and Maps really plays that part.” 

Google’s closely guarded “Ground Truth” project, initiated in 2008 but only publicly revealed just 
four months after the FCC report in 2012, exemplifies the point. Ground Truth is the “deep map” that 
contains the detailed “logic of places”: walking paths, goldfish ponds, highway on-ramps, traffic 
conditions, ferry lines, parks, campuses, neighborhoods, buildings, and more.®! Getting these details right 
is a source of competitive advantage in the contest for behavioral surplus accrued from mobile devices. 
The construction of the deep map draws on public resources such as geographic databases from the US 
Census Bureau and the US Geological Survey, but what distinguishes Google’s maps from all others is 


the integration of its exclusive proprietary data from Street View. In other words, data compiled through 
public investments are augmented with data taken from a unilateral transfer of surplus behavior and 
decision rights. The composite results are then reclassified as private assets. 

One of the first journalists invited to see demonstrations of Ground Truth in 2012, Alexis Madrigal, 
observed that “the Maps team, largely driven by Street View, is publishing more imagery data every two 
weeks than Google possessed in total in 2006.... Google is up to five million miles driven now.” Street 
View cars are likened to Google Search’s early web crawlers, which quietly commandeered web pages 
for indexing and access in the corporation’s original act of dispossession. By 2012, Street View data also 
provided street signs and addresses. Soon, Madrigal wrote, “any word that is visible from a road will 
become a part of Google’s index of the physical world” thanks to Street View. Madrigal’s look at the 
Ground Truth operation concludes, “The geographic data Google has assembled is not likely to be 
matched by any other company.... They’ve built this whole playground as an elaborate lure for you.”™ 

As one project leader put it, “The challenge of deciding you’re going to map the world is that you 
can’t ever stop.” So it was that by 2016, Google’s Street View website celebrated its successful 
evolution by stating, “We’ve come a long way since our initial U.S. launch in 2007; today we’ve 
expanded our 360-degree panoramic views to include locations on all seven continents.” Street View’s 
fleet of surveillance-gathering tools had been augmented to include a wearable backpack, a three-wheeled 
pedicab, a snowmobile, and a trolley, all of which were designed to capture places that Street View cars 
could not traverse. Tourist boards and nonprofits were offered the use of the company’s Trekker 
equipment (the backpack camera) to “collect views of remote and unique places” that were, literally and 
figuratively, “off the grid.”© 

What Google couldn’t build, it bought. In 2013 the corporation won a reported bidding war with 
Facebook for Israeli social mapping startup Waze, a firm that pioneered community-sourced real-time 
traffic information. In 2014 it acquired real-time satellite imaging startup Skybox just as the US 
Department of Commerce lifted restrictions on high-resolution satellite imagery. As an expert explained, 


If you imagine a satellite above your office then the old resolution could probably make out your desk. The new imagery—where 


each pixel measures around 31 cm—can now make out what’s on your desk. When you reach this sort of frequency you can begin to 


add in what we call “pattern of life” analysis. This means looking at activity in terms of movement—not just identification. © 


In this context one appreciates the significance of another aspect of Google’s redirection campaign: a 
2011 announcement that the corporation had breached “a new frontier” with the introduction of an “indoor 
positioning system” that enabled it to locate and follow people “when you’re inside an airport, shopping 
mall, or retail store.” Eventually, these new capabilities would include sensors and embedded cameras 
that let users map and navigate interior spaces. In a September 2014 blog, Google Maps’ dynamic new 
capabilities were showcased to the public as your new “co-pilot for deciding everything from turn-by- 
turn directions, to discovering new restaurants, to deciding which hiking trails to climb.” The post credits 
Street View with providing these wondrous new capabilities and announces the expansion of the whole 
incursion with the introduction of a mobile mapping tool dubbed “Cartographer,” worn as a backpack and 
able to map the interior of buildings. Cartographer’s information could be added to the growing 
navigational database of interior spaces, amplifying Google’s ability to locate people and devices as they 
moved between outdoor and indoor spaces. 

Building interiors had eluded Street View and the extraction imperative; few homeowners were likely 
to invite those cameras indoors. Instead, Cartographer’s capabilities were bundled into the larger Street 
View redirection campaign and pitched to businesses as a way to enhance consumer trust, allay anxiety, 


and substantially increase revenues. Google exhorted consumer-facing businesses to “invite customers 
inside.” With its “Business View,” consumers would be able to see inside thousands of hotels, 
restaurants, and other destinations. Search listings would feature the new Street View content. Hotel 
listings would offer a virtual tour of the properties. “Give them the confidence they’re seeking,” Google 
told its business marketplace, by allowing consumers “to experience your location before they arrive.” 
Google asserted that virtual tours “double bookings,” and it instated a certification program that enabled 
businesses to hire a Google-approved freelance photographer to produce images for Street View. These 
extraordinary new redirection tactics aimed to flip the old pattern. They reframed Street View from an 
edgy incursion circumventing resistance through stealth to an opulent VIP tent where businesses 
scrambled for an entry pass. 

Street View’s redirection and elaboration announced a critical shift in the orientation and ambition of 
the surveillance program: it would no longer be only about routes, but about routing. We will examine 
this new episode of dispossession in the chapters that follow. For now, suffice to say that Street View and 
the larger project of Google Maps illustrate the new and even more ambitious goals toward which this 
cycle of dispossession would soon point: the migration from an online data source to a real-world 
monitor to an advisor to an active shepherd—from knowledge to influence to control. Ultimately, Street 
View’s elaborate data would become the basis for another complex of spectacular Google incursions: the 
self-driving car and “Google City,” which we learn more about in Chapter 7. Those programs aim to take 
surplus capture to new levels while opening up substantial new frontiers for the establishment of 
behavioral futures markets in the real world of goods and services. It is important to understand that each 
level of innovation builds on the one before and that all are united in one aim, the extraction of behavioral 
surplus at scale. 

In this progression, Google perceives an opportunity that it hopes its customers will come to 
appreciate: its ability to influence actual behavior as it occurs in the real spaces of everyday life. In 2016, 
for example, the corporation introduced a new Maps app feature called “Driving Mode” that suggests 
destinations and travel times without users even selecting where they want to go. If you searched for a 
hammer online, then “Driving Mode” can send you to a hardware store when you buckle up your seat belt. 
“Google is integrating this ‘push’ technology into its main mobile search app,” reported the Wall Street 
Journal. 

With this app, Google the “copilot” prompts an individual to turn left and right on a path defined by its 
continuously accruing knowledge of the person and the context. Predictions about where and why a 
person might spend money are derived from Google’s exclusive access to behavior surplus and its 
equally exclusive analytic capabilities: “Eat here.” “Buy this.” Google’s surplus analysis can predict 
that you are likely to buy an expensive woolen suit, and its real-time location data can trigger the 
proprietor or advertiser’s real-time prompt, matched to your profile and delivered at the very moment that 
you are within sight of the flannels, tweeds, and cashmeres. Push and pull, suggest, nudge, cajole, shame, 
seduce: Google wants to be your copilot for life itself. Each human response to each commercial prompt 
yields more data to refine into better prediction products. The prompts themselves are bought and paid for 
in a novel iteration of Google’s online ad markets: real-time, real-world trading in behavioral futures. 
Your future. 

The stakes are high in this market frontier, where unpredictable behavior is the equivalent of lost 
revenue. Google cannot leave anything to chance.” In September 2016 the tech newsletter the Register 
revealed that the Google Play app preinstalled in the latest Android phone continuously checks a user’s 
location, sending that information to your third-party apps as well as to Google’s own servers. One 
security researcher was shocked when his Android phone prompted him to download the McDonald’s 


app at the very moment that he crossed the threshold of the fast-food restaurant. He later discovered that 
Google Play had monitored his location thousands of times. Similarly, Google Maps “doesn’t give you a 
decent option of turning it off.” If you do, the operating system warns, “basic features of your device may 
no longer function as intended.” Google’s insistence reflects the authoritarian politics of the extraction 
imperative as well as the corporation’s own enslavement to the implacable demands of its economics. 

The historic point for us to consider here is that the once-spurned Street View found new life in its 
contribution to the decisive expansion of behavioral futures markets both online and in the real world. 
Once dedicated to targeted online advertising, these markets now grow to encompass predictions about 
what human beings will do now, soon, and later, whether they make their way online, on sidewalks and 
roads, or through rooms, halls, shops, lobbies, and corridors. These ambitious goals foreshadow fresh 
incursions and dispossessions as resistance is neutralized and populations fall into dulled submission. 

Google discovered by chance or intention the source of every mapmaker’s power. The great historian 
of cartography, John B. Harley, said it succinctly: “Maps created empire.” They are essential for the 
effective “pacification, civilization, and exploitation” of territories imagined or claimed but not yet seized 
in practice. Places and people must be known in order to be controlled. “The very lines on the map,” 
wrote Harley, were a language of conquest in which “the invaders parcel the continent among themselves 
in designs reflective of their own complex rivalries and relative power.” The first US rectangular land 
survey captured this language perfectly in its slogan: “Order upon the Land.” The cartographer is the 
instrument of power as the author of that order, reducing reality to only two conditions: the map and 
oblivion. The cartographer’s truth crystallizes the message that Google and all surveillance capitalists 
must impress upon all humans: if you are not on our map, you do not exist. 


IV. The Dogs of Audacity 


Projects such as Street View taught Google that it could assume the role of arbiter of the future and get 
away with it. It learned to sustain even the most-contested dispossession efforts when they are necessary 
to secure vital new supply lines. For example, while Street View protests erupted around the world and 
just months before Germany’s announcement that Street View was secretly capturing personal information 
from unprotected Wi-Fi networks, Google introduced Buzz—a platform intended to float Google’s nets in 
the path of the coveted behavioral surplus that streamed from social networks. The invasive practices 
introduced with Buzz—it commandeered users’ private information to establish their social networks by 
fiat—set off a fresh round of the dispossession cycle and its dramatic contests. 

As Google learned to successfully redirect supply routes, evading and nullifying opposition, it became 
even more emboldened to let slip the dogs of audacity and direct them toward havoc. Among many 
examples, Google Glass neatly illustrates the tenacity of the extraction imperative and its translation into 
commercial practice. Google Glass combined computation, communication, photography, GPS tracking, 
data retrieval, and audio and video recording capabilities in a wearable format patterned on eyeglasses. 
The data it gathered—location, audio, video, photos, and other personal information—moved from the 
device to Google’s servers, merging with other supply routes to join the titanic one-way flow of 
behavioral surplus. 

The project was seen as a precursor to more flexible and less overt forms of wearable computation 
and surplus capture. John Hanke described its familiar shape in the form of eyewear as suitable for “the 
early adoption phases” of wearable technology in much the same way that the first automobiles resembled 
horse-drawn buggies. In other words, the “glasses” were intended to disguise what was in fact 


unprecedented: “Ultimately we will want these technologies, wherever they are on your body, to be 
totally optimized based on the job they’re doing, not on what is more socially acceptable at that first 
moment of creation, just because it reminds people of something they’ve seen in the past.” Introduced 
with great flair in the spring of 2012 as fashion-forward futurism, it wasn’t long before the public 
registered fresh horror at this bizarre invasion. Those who wore the device were recast as “glassholes,” 
and some businesses banned the glasses from their premises.“ 

Privacy advocates protested the “always on” but “undetectable” recording of people and places that 
eliminates a person’s reasonable expectation of privacy and/or anonymity. They warned of new risks as 
facial-recognition software is applied to these new data streams and predicted that technologies like 
Glass would fundamentally alter how people behave in public. By May 2013, a congressional privacy 
caucus asked CEO Larry Page for assurances on privacy safeguards for Glass, even as a Google 
conference was held to coach developers on creating apps for the new device. In April 2014 Pew 
Research announced that 53 percent of Americans thought that smart wearables were “a change for the 
worse,” including 59 percent of American women. 

Google continued to tough it out, waiting for habituation to kick in. That June it announced that Glass 
would offer the Livestream video-sharing app, enabling Glass users to stream everything around them to 
the internet in real time. When asked about these controversial and intrusive capabilities in the hands of 
any owner of the device, Livestream’s CEO reckoned, “Google is ultimately in charge of... setting the 
rules.”“2 Sergey Brin made it clear that any resistance would be categorically rejected when he told the 
Wall Street Journal, “People always have a natural aversion to innovation.” 

Adaptation began in 2015 with the announcement that Glass would no longer be available. The 
company said nothing to acknowledge the public’s revulsion or the social issues that Glass had raised. A 
short blog post announced, “Now we’re ready to put on our big kid shoes and learn how to run... you’ll 
start to see future versions of Glass when they’re ready.” An eyewear designer was tasked with 
transforming the look from a futuristic device to something more beautiful. 

Redirection began quietly. In June 2015 the FCC’s Office of Engineering and Technology received 
new design plans for Glass, and September brought fresh headlines announcing that Glass “is getting a 
new name, and a new lease on life.” A year later, Eric Schmidt, now Google’s chairman, put the 
situation into perspective: “It is a big and very fundamental platform for Google.” He explained that Glass 
was withdrawn from public scrutiny only to “make it ready for users... these things take time.” As more 
information trickled out of the corporation, it became clear that there was no intention of ceding potential 
new supply routes in wearable technologies, no matter the public reaction. Glass was the harbinger of a 
new “wearables” platform that would help support the migration of behavioral surplus operations from 
the online to the offline world.®! 

In July 2017 the redirection phase went public with a blog post introducing a new iteration of Google 
Glass to the world, now as “Glass Enterprise Edition.”®* This time there would be no frontal attack on 
public space. Instead, it was to be a tactical retreat to the workplace—the gold standard of habituation 
contexts, where invasive technologies are normalized among captive populations of employees. “Workers 
in many fields, like manufacturing, logistics, field services, and healthcare find it useful to consult a 
wearable device for information and other resources while their hands are busy,” wrote the project’s 
leader, and most press accounts lauded the move, citing productivity and efficiency increases in factories 
that deployed the new Glass.®2 There was little acknowledgment that habituation to Glass at work was 
most certainly a back door to Glass in our streets or that the intrusive surveillance properties of the 
device would, with equal certainty, be imposed on the women and men required to use them as a 
condition of their employment. 


The lesson of Glass is that when one route to a supply source encounters obstacles, others are 
constructed to take up the slack and drive expansion. The corporation has begrudgingly learned to pay 
more attention to the public relations of these developments, but the unconditional demands of the 
extraction imperative mean that the dispossession cycle must proceed at full throttle, continuously 
claiming new territory. 

Dispossession may be an act of “simple robbery” in theory, but in fact it is a complex, highly 
orchestrated political and material process that exhibits discernible stages and predictable dynamics. The 
theory of change exhibited here systematically transfers knowledge and rights from the many to the few in 
a glorious fog of Page’s “automagic.” It catalogues public contest as the unfortunate but predictable outcry 
of foolish populations who exhibit a knee-jerk “resistance to change,” wistfully clinging to an 
irretrievable past while denying an inevitable future: Google’s future, surveillance capitalism’s future. 
The theory indicates that opposition must simply be weathered as the signature of the first difficult phases 
of incursion. It assumes that opposition is fleeting, like the sharp yelp of pain when a Novocain needle 
first pierces the flesh, before numbness sets in. 


V. Dispossession Competition 


Google’s spectacular success in constructing the mechanisms and principles of surveillance capitalism 
and attracting surveillance revenues ignited competition in an escalating war of extraction. Google began 
in a blank space, but it would soon contend with other firms drawn to surveillance revenues. Facebook 
was the first and has remained the most aggressive competitor for behavioral surplus supplies, initiating a 
wave of incursions at high speed, establishing a presence on the free and lawless surplus frontier while 
denying its actions, repelling criticism, and thoroughly confusing the public. The “Like” button, introduced 
widely in April 2010 as a communications tool among friends, presented an early opportunity for 
Facebook’s Zuckerberg to master the dispossession cycle. By November of that year, a study of the 
incursion already underway was published by Dutch doctoral candidate and privacy researcher Arnold 
Roosendaal, who demonstrated that the button was a powerful supply mechanism from which behavioral 
surplus is continuously captured and transmitted, installing cookies in users’ computers whether or not 
they click the button. Presciently describing the operation as an “alternative business model,” Roosendaal 
discovered that the button also tracks non-Facebook members and concluded that Facebook was 
potentially able to connect with, and therefore surveil, “all web users.”84 Only two months earlier, 
Zuckerberg had characterized Facebook’s growing catalogue of privacy violations as “missteps.”®2 Now 
he stuck to the script, eventually calling Roosendaal’s discovery a “bug.” °° 

By 2011 the habituation stage of the cycle was in full swing. A May Wall Street Journal report 
confirmed Facebook’s tracking, even when users don’t click the button, and noted that the button was 
already installed on one-third of the world’s one thousand most-visited websites. Meanwhile, Facebook’s 
Chief Technology Officer said of the button, “We don’t use them for tracking and they’re not intended for 
tracking.”®2 On September 25, Australian hacker Nik Cubrilovic published findings showing that 
Facebook continued to track users even after they logged out of the site.2° Facebook announced that it 
would fix “the glitch,” explaining that certain cookies were tracking users in error, and noting that it could 
not cease the practice entirely due to “safety” and “performance” considerations.®? Journalists discovered 
that just three days before Cubrilovic’s revelations, the corporation received a patent on specialized 
techniques for tracking users across web domains. The new data methods enabled Facebook to track 
users, create personal profiles on individuals and their social networks, receive reports from third parties 


on each action of a Facebook user, and log those actions in the Facebook system in order to correlate them 
with specific ads served to specific individuals.22 The company immediately denied the relevance and 
importance of the patent.2! 

With Facebook’s unflinching assertions that it did not track users, even in the face of many robust facts, 
specialists grew more frustrated and the public more confused. This appears to have been the point. By 
denying every accusation and pledging its commitment to user well-being, Facebook secured a solid year 
and a half with which to habituate the world to its “Like” button, institutionalizing that iconic thumb turned 
toward the sky as an indispensable prosthetic of virtual communication. 

This solid achievement paved the way for the adaptation stage of the dispossession cycle, when in 
late November 2011, Facebook consented to a settlement with the FTC over charges that it had 
systematically “deceived consumers by telling them that they could keep their Facebook information 
private, and then repeatedly allowing it to be shared and made public.”22 The complaint brought by EPIC 
and a coalition of privacy advocates in 2009 initiated an FTC investigation that yielded plenty of 
evidence of the corporation’s broken promises. These included website changes that made private 
information public, third-party access to users’ personal data, leakage of personal data to third-party 
apps, a “verified apps” program in which nothing was verified, enabling advertisers to access personal 
information, allowing access to personal data after accounts were deleted, and violations of the Safe 
Harbor Framework, which governs data transfers between the United States and the EU. In the parallel 
universe of surveillance capitalism, each one of these violations was worthy of a five-star rating from the 
extraction imperative. The FTC order barred the company from making further privacy 
misrepresentations, required users’ affirmative consent to new privacy policies, and mandated a 
comprehensive privacy program to be audited every two years for twenty years. FTC Chairman Jon 
Leibowitz insisted that “Facebook’s innovation does not have to come at the expense of consumer 
privacy.”2 But Leibowitz was not up against a company; he was up against a new market form with 
distinct and intractable imperatives whose mandates can be fulfilled only at the expense of user privacy. 

Redirection came swiftly. In 2012 the company announced it would target ads based on mobile app 
use, as it worked with Datalogix to determine when online ads result in a real-world purchase. This 
gambit required mining personal information, including e-mail addresses, from user accounts. In 2012 
Facebook also gave advertisers access to targeting data that included users’ e-mail addresses, phone 
numbers, and website visits, and it admitted that its system scans personal messages for links to third- 
party websites and automatically registers a “like” on the linked web page.2° By 2014, the corporation 
announced that it would be tracking users across the internet using, among its other digital widgets, the 
“Like” button, in order to build detailed profiles for personalized ad pitches. Its “comprehensive privacy 
program’ advised users of this new tracking policy, reversing every assertion since April 2010 with a 
few lines inserted into a dense and lengthy terms-of-service agreement. No opt-out privacy option was 
offered.2’ The truth was finally out: the bug was a feature. 

Meanwhile, Google maintained the pledge that had been critical to the FTC’s approval of its 2001 
acquisition of the ad-tracking behemoth DoubleClick when it agreed not to combine data from the tracking 
network with other personally identifiable information in the absence of a user’s opt-in consent. In this 
case, Google appears to have waited for Facebook to extend the surveillance capitalist frontier and bear 
the brunt of incursion and habituation. Later, in the summer of 2016, Google crossed that frontier with an 
announcement that a user’s DoubleClick browsing history “may be” combined with personally 
identifiable information from Gmail and other Google services. Its promised opt-in function for this new 
level of tracking was presented with the headline “Some new features for your Google account.” One 
privacy scholar characterized the move as the final blow to the last “tiny semblance” of privacy on the 


web. A coalition of privacy groups presented a new complaint to the FTC, implicitly recognizing the 
logic of the dispossession cycle: “Google has done incrementally and furtively what would plainly be 
illegal if done all at once.” 

Facebook’s IPO in 2012 was notoriously botched when last-minute downward revisions of its sales 
projections, precipitated by the rapid shift to mobile devices, led to some unsavory dealings among its 
investment bankers and their clients. But Zuckerberg, Sheryl Sandberg, and their team quickly mastered 
the nuances of the dispossession cycle, this time to steer the company toward mobile ads. They learned to 
be skilled and ruthless hunters of behavioral surplus, capturing supplies at scale, evading and resisting 
law, and upgrading the means of production to improve prediction products. 

Surveillance revenues flowed fast and furiously, and the market lavishly rewarded the corporation’s 
shareholders. By 2017, the Financial Times hailed the company’s 71 percent earnings surge with the 
headline “Facebook: The Mark of Greatness” as Facebook’s market capitalization rose to just under $500 
billion, with 2 billion average monthly active users. Facebook ranked seventh in one important tally of the 
top 100 companies in the first quarter of 2017, when just a year earlier it hadn’t figured anywhere in the 
top 100. Advertising, primarily mobile, accounted for nearly every dollar of the company’s revenue in the 
second quarter of 2017: $9.2 billion of a total $9.3 billion and a 47 percent increase over the prior year. 

The Guardian reported that Google and Facebook accounted for one-fifth of global ad spending in 
2016, nearly double the figure of 2012, and by one accounting Google and Facebook owned almost 90 
percent of the growth in advertising expenditures in 2016.42 Surveillance capitalism had propelled these 
corporations to a seemingly impregnable position. 

Among the remaining three of the largest internet companies, Microsoft, Apple, and Amazon, it was 
Microsoft that first and most decisively turned toward surveillance capitalism as the means to restore its 
leadership in the tech sector, with the appointment of Satya Nadella to the role of CEO in February 2014. 
Microsoft had notoriously missed several key opportunities to compete with Google in the search 
business and develop its targeted advertising capabilities. As early as 2009, when Nadella was a senior 
vice president and manager of Microsoft’s search business, he publicly criticized the company’s failure to 
recognize the financial opportunities associated with that early phase of surveillance capitalism. “In 
retrospect,” he lamented, “it was a terrible decision” to end the search-ad service: “None of us saw the 
paid-search model in all its glory.” Nadella recognized then that Microsoft’s Bing search engine could not 
compete with Google because it lacked scale in behavioral surplus capture, the critical factor in the 
fabrication of high-quality prediction products: “When you look at search... it’s a game of scale. Clearly 
we don’t have sufficient scale today and that hinders... the quality of the ad relevance which is perhaps 
the bigger issue we have today.”12! 

Less than three months after assuming his new role, Nadella announced his intention to redirect the 
Microsoft ship straight into this game of scale with the April release of a study that the company had 
commissioned from market intelligence firm IDC.1% It concluded that “companies taking advantage of 
their data have the potential to raise an additional $1.6 trillion in revenue over companies that don’t,” and 
Nadella was determined to make landfall on the far shores of this rich new space. Microsoft would reap 
the advantages of its own data, and it would specialize in “empowering” its clients to do the same. 
Nadella composed a blog to signal the new direction, writing, “The opportunity we have in this new 
world is to find a way of catalyzing this data exhaust from ubiquitous computing and converting it into fuel 
for ambient intelligence.”/22 As a video outlining the new “data vision” explains, “Data that was once 
untapped is now an asset.” 

Many of Nadella’s initiatives aim to make up for lost time in establishing robust supply routes to 
surplus behavior and upgrading the company’s means of production. Bing’s search engineering team built 


its own model of the digital and physical world with a technology it calls Satori: a self-learning system 
that is adding 28,000 DVDs of content every day. According to the project’s senior director, “It’s mind- 
blowing how much data we have captured over the last couple of years. The line would extend to Venus 
and you would still have 7 trillion pixels left over.”/22 All those pixels were being put to good use. In its 
October 2015 earnings call, the company announced that Bing had become profitable for the first time, 
thanks to around $1 billion in search ad revenue from the previous quarter. 

Another strategy to enhance Bing’s access to behavioral surplus was the corporation’s “digital 
assistant,” Cortana, to which users addressed more than one billion questions in the three months after its 
2015 launch.22 As one Microsoft executive explains, “Four out of five queries go to Google in the 
browser. In the [Windows 10] task bar [where Cortana is accessed, ] five out of five queries go to Bing.... 
We’re all in on search. Search is a key component to our monetization strategy.” 12/ 

Cortana generates more than search traffic. As Microsoft’s privacy policy explains, “Cortana works 
best when you sign in and let her use data from your device, your personal Microsoft account, other 
Microsoft services, and third-party services you choose to connect.” 128 Like Page’s automagic, Cortana is 
intended to inspire awestruck and grateful surrender. One Microsoft executive characterizes Cortana’s 
message: “‘I know so much about you. I can help you in ways you don’t quite expect. I can see patterns 
that you can’t see.’ That’s the magic.” 12 

Nevertheless, the company made a canny decision not to disclose the true extent of Cortana’s 
knowledge to its users. It wants to know everything about you, but it does not want you to know how much 
it knows or that its operations are entirely geared to continuously learning more. Instead, the “bot” is 
programmed to ask for permission and confirmation. The idea is to avoid spooking the public by 
presenting Cortana’s intelligence as “progressive” rather than “autonomous,” according to the project’s 
group program manager, who noted that people do not want to be surprised by how much their phones are 
Starting to take over: “We made an explicit decision to be a little less ‘magical’ and little more 
transparent.” 

Nadella envisions a new platform of “conversations” in which users interact with bots that induce 
them to eagerly disclose the details of their daily lives! The platform promises to deliver experiences 
such as “conversational commerce,”2 where, for example, a bot 


knows what shoes you bought last week, it knows your preferences from your past purchases, it knows your profile and can call a 
recommendations model to determine what products you have the most affinity to buy.... Using the power of data and analytics, the 
bot can respond back with recommendations that it determines are most relevant for you. It can also invite people from your social 


network to help you make a choice. Once you make the selection, it will use your size information, shipping address, payment 


information to ship the selected dress to you,113 


The release of Microsoft’s new operating system, Windows 10, in July 2015 drove home the 
seriousness of purpose and urgency that the corporation now assigned to establishing and securing supply 
routes to behavioral surplus.44 One software engineer writing in Slate described it as “a privacy morass 
in dire need of reform” as he detailed how the system “gives itself the right to pass loads of your data to 
Microsoft’s servers, use your bandwidth for Microsoft’s own purposes, and profile your Windows 
usage.” 19 

As many analysts quickly discovered, the system pushed users toward the “express install” function, in 
which every default setting enabled the maximum flow of personal information to the corporation’s 
servers. An investigation by tech website Ars Technica revealed that even when those default settings 
were reversed and key services such as Cortana disabled, the system continued to access the internet and 


transmit information to Microsoft. In some instances those transmissions appeared to contain personal 
information, including a machine ID, user content, and location data Us 

According to an analysis by the Electronic Frontier Foundation (EFF), even users who opted out of 
Cortana were subject to an “unprecedented” amount of information capture, including text, voice, and 
touch input; web tracking; and telemetry data on their general use, programs, session duration, and more. 
The EFF also found that the company chose to hold security functions hostage to personal data flows, 
claiming that security updates for the operating system would not function properly if users chose to limit 
location reporting.” 

In 2016 Microsoft acquired LinkedIn, the professional social network, for $26.2 billion. The aim here 
is to establish reliable supply routes to the social network dimension of surplus behavior known as the 
“social graph.” These powerful new flows of social surplus from 450 million users can substantially 
enhance Microsoft prediction products, a key fact noted by Nadella in his announcement of the acquisition 
to investors: “This can drive targeting and relevance to the next level.”4!8 Of the three key opportunities 
that Nadella cited to investors upon the announcement of the acquisition, one was “Accelerate 
monetization through individual and organization subscriptions and targeted advertising.” Among the key 
factors here would be unified professional profiles across all services, devices, and channels and 
Microsoft’s comprehensive knowledge of each individual user: “Today Cortana knows about you, your 
organization and about the world. In the future, Cortana will also know your entire professional network 
to connect dots on your behalf and stay one step ahead.” 2 

Once again, the market richly rewarded Microsoft, and Nadella, for the pivot toward surveillance 
revenues. When Nadella climbed into the CEO’s chair in February 2014, the company’s shares were 
trading at around $34, and its market value was roughly $315 billion. Three years later, in January 2017, 
the corporation’s market capitalization topped $500 billion for the first time since 2000, and its shares 
rose to an all-time high of $65.64.122 


VI. The Siren Song of Surveillance Revenues 


The unprecedented successes of Google, Facebook, and then Microsoft exerted a palpable magnetism on 
the global economy, especially in the US, where the politics of lawlessness were most firmly entrenched. 
It did not take long before companies from established sectors with roots far from Silicon Valley 
demonstrated their determination to compete for surveillance revenues. Among the first in this second 
wave were the telecom and cable companies that provide broadband service to millions of individuals 
and households. Although there is some debate about whether these companies can effectively compete 
with the established internet giants, the facts on the ground suggest that the ISPs are nonetheless 
determined to try. “Armed with their expansive view over the entire web, internet providers may even be 
in a position to out-Facebook Facebook, or out-Google Google,” observed the Washington Post. 

The largest of these corporations—Verizon, AT&T, and Comcast—made strategic acquisitions that 
signaled a shift away from their long-standing models of fees for service in favor of monetizing 
behavioral surplus. Their tactical maneuvers demonstrate the generalizability of surveillance capitalism’s 
foundational mechanisms and operational requirements, and are evidence that this new logic of 
accumulation defines a wholly new territory of broad-based market endeavor. 

Verizon—the largest telecom company in the US and the largest in the world as measured by market 
capitalization‘2*—publicly introduced its shift toward surveillance revenues in the spring of 2014, when 
an article in Advertising Age announced the company’s move into mobile advertising. Verizon’s VP of 


data marketing argued that such advertising had been limited by “addressability... the growing difficulty 
of tracking consumers as they move between devices.” As one marketing expert complained, “There isn’t 
a pervasive identity that tracks users from mobile applications and your mobile browser.” The article 
explained that Verizon had developed “a cookie alternative for a marketing space vexed by the absence of 
cookies.” Verizon aimed to solve advertisers’ tracking needs by assigning a hidden and undeletable 
tracking number, called a PrecisionID, to each Verizon user.223 

In fact, Verizon’s incursion had launched two years earlier in 2012 but had been carefully hidden from 
the public. That was probably because the ID enables the corporation to identify and monitor individuals’ 
habits on their smartphones and tablets, generating behavioral surplus while bypassing customers’ 
awareness. The tracker can neither be turned off nor evaded with private browsing or other privacy tools 
and controls. Whenever a Verizon subscriber visits a website or mobile app, the corporation and its 
partners use this hidden ID to aggregate and package behavioral data, all without customers’ knowledge. 

Verizon’s indelible tracking capabilities provided a distinct advantage in the growing competition for 
behavioral surplus. Advertisers hungry to redefine your walk in the park as their “marketing space” could 
now reliably target ads to your phone on the strength of the corporation’s indelible personal identifier. 
Verizon also entered into partnership with Turn, an advertising technology firm already notorious for the 
invention of an unusual “zombie cookie” or “perma-cookie” that immediately “respawns” when a user 
chooses to opt out of ad tracking or deletes tracking cookies. As a Verizon partner, the Turn zombie 
cookie attached itself to Verizon’s secret tracking number, adding even more protection from discovery 
and scrutiny. Turn’s chief “privacy officer” defended the arrangement, saying, “We are trying to use the 
most persistent identifier that we can in order to do what we do.”124 

By the fall of 2014, Verizon’s stealthy new claim on free raw material was outed by Jacob Hoffman- 
Andrews, a technologist with the Electronic Frontier Foundation. An article in Wired called attention to 
Hoffman-Andrews’ analysis of Verizon’s surveillance program and his additional discovery that AT&T 
was using a similar tracking ID. The article quoted a Verizon spokesperson admitting, “There’s no way to 
turn it off.”42° Hoffman-Andrews observed that even when customers opt out of Verizon’s targeted ads, its 
tracking ID persists, as the corporation bypasses or overrides all signals of a user’s intentions, including 
the Do Not Track setting, Incognito and other private browsing modes, and cookie deletion. The ID is then 
broadcast to every “unencrypted website a Verizon customer visits from a mobile device. It allows third- 
party advertisers and websites to assemble a deep, permanent profile of visitors’ web browsing habits 
without their consent.”2° Alarmed by the threat of fresh competition, Google, posing as a privacy 
advocate, launched a campaign for a new internet protocol that would prevent “header injections” such as 
Verizon’s PrecisionID.1~ 

Privacy expert and journalist Julia Angwin and her colleagues at ProPublica reported that similar 
tracking IDs were becoming standard throughout the telecom industry. As one ad executive put it, “What 
we’re excited about is the carrier-level ID, a higher-level recognition point that lets us track with 
certainty....” Hoffman-Andrews would eventually call the telecom’s tactics “a spectacular violation of 
Verizon users’ privacy.”/28 True as this may be, the corporation’s tactical operations suggest an even more 
far-ranging development. 

Verizon would not retreat from the territory already claimed with its incursion. The hidden ID would 
stay, and the company assured customers that “it is unlikely that sites and ad entities will attempt to build 
customer profiles.”/“2 However, it didn’t take long for experts to discover that Twitter’s mobile 
advertising arm already relied on the Verizon ID to track Twitter users’ behavior.42? Then computer 
scientist and legal scholar Jonathan Mayer found that Turn’s zombie cookie sent and received data from 
more than thirty businesses, including Google, Facebook, Yahoo!, Twitter, Walmart, and WebMD. Mayer 


investigated both Verizon and Turn’s opt-out policies and found both to be deceptive, concluding that 
every one of Verizon’s public statements on the privacy and security of its tracking ID was false. “For an 
ordinary user,” he wrote, “there simply is no defense.” 12 

Verizon’s substantial entry into surveillance capitalism necessarily tethered the corporation’s interests 
to the extraction imperative. We can see this in the way that Verizon discovered and implemented the 
dispossession cycle, moving rapidly through its sequence of tactical phases from incursion to redirection. 
Verizon’s initial incursion bought it three years of internal experimentation and discovery. During that time 
it crossed the threshold of public awareness, beginning the gradual process of public habituation to its 
new practices. Once its strategies were public, it endured a barrage of critical news articles and the 
scrutiny of privacy experts, but it also bought more time to explore revenue opportunities and supply route 
expansion. Public reaction to its incursion forced the corporation to map the next phases of the cycle. 

Public pressure triggered the shift toward adaptation in early 2015. An FCC investigation into 
Verizon’s illicit tracking practices had been launched a few months earlier. The Electronic Privacy 
Information Center circulated a petition in January 2015 demanding that the FCC penalize the company. 
By the end of that month, the Senate Committee on Commerce, Science, and Transportation published a 
letter to Verizon expressing “deep concern” over its new practices.4°* The committee chastised Verizon 
and Turn for their “seemingly” deliberate “violation of consumer privacy” and “circumvention of 
customer choice.”/22 Within a day of the letter’s publication, Verizon announced, “We have begun working 
to expand the opt-out to include the identifier referred to as the UIDH [unique identifier header], and 
expect that to be available soon.” The New York Times called Verizon’s announcement “a major revision 
of its mobile ad-targeting program.”4 

The Times could not have known that the redirection phase of the dispossession cycle was already in 
motion. In May 2015 Verizon agreed to purchase AOL for $4.4 billion. As many analysts quickly 
appreciated, the real attraction of AOL was its CEO, Tim Armstrong, the first head of advertising sales at 
Google and the man who oversaw its transition from Madison Avenue-style advertising to AdWords’ 
breakthrough discoveries. He was president of Google’s Americas sales division when, like Sheryl 
Sandberg before him, Armstrong left Google for AOL in 2009 with a profound grasp of AdWords’ 
surveillance DNA and the determination to rescue AOL’s balance sheet with surveillance capitalism gene 
therapy. As Verizon’s president of Operations told investors, “For us, the principal interest was around 
the ad tech platform that Tim Armstrong and his team have done a really terrific job building.” Forbes 
observed that Armstrong needed Verizon’s resources “to challenge the duopoly of Google and 
Facebook.”!22 

Any serious challenge to the giant surveillance capitalists must begin with economies of scale in 
behavioral surplus capture. To that end, Verizon immediately redirected its supply routes through AOL’s 
advertising platforms. Within a few months of the acquisition, Verizon quietly posted a new privacy notice 
on its website that few of its 135 million wireless customers would ever read. A few lines slipped into 
the final paragraphs of the post told the story: PrecisionID is on the move again. Verizon and AOL would 
now work together “to deliver services that are more personalized and useful to you... we will combine 
Verizon’s existing advertising programs... into the AOL Advertising Network. The combination will help 
make the ads you see more valuable across the different devices and services you use.” The new notice 
asserted that “the privacy of our customers is important to us,” though not important enough to 
compromise the extraction imperative and allow raw-material providers to challenge the corporation’s 
dispossession program. Opt-out procedures were available but, as usual, complex, difficult to ascertain, 
and time-consuming. “Please note,” the post concluded, “that using browser controls such as clearing 
cookies on your devices or clearing your browser history is not an effective way to opt out of the Verizon 


or AOL advertising programs.” 12° 


The FCC settlement with Verizon was another gloomy example of a public institution outmatched by 
the velocity and resources of a determined surveillance capitalist. In March 2016, long after the 
announcement of Verizon’s tactical redirection, the FCC reached a $1.35 million settlement with Verizon 
over its hidden ID privacy violations. Although Verizon agreed to relaunch its cookie on an opt-in basis, 
the settlement did not extend to AOL's advertising network, which is where the action had moved. 
Verizon’s burgeoning new supply routes would remain unchallenged.’ Later that month, Armstrong 
would meet with ad buyers, a rendezvous described by the Wall Street Journal as “his first real chance to 
pitch that AOL—fresh off its sale to Verizon Communications Inc.—intended to become a credible threat 
to Facebook Inc. and Google....”428 

On March 31, 2016, the FCC issued a Notice of Proposed Rulemaking that would establish privacy 
guidelines for ISPs. The companies would be allowed to continue to collect behavioral data that 
enhanced the security and effectiveness of their own services, but all other uses of “consumer data” 
would require opt-in consent. “Once we subscribe to an ISP,” FCC Chairman Tom Wheeler wrote, “most 
of us have little flexibility to change our mind or avoid that network rapidly.”/“2 The proposal was aimed 
exclusively at ISPs, considered under the jurisdiction of the FCC, but did not include internet companies, 
which the Federal Trade Commission is charged with regulating. 

In light of the high-stakes dispossession competition already under way among key ISPs, it is not 
surprising that the proposal quickly became a political lightning rod. ISPs, their lobbyists, policy 
advisors, and political allies lined up to kill the effort, stating that ISPs’ competitive prospects would be 
unfairly impeded: “Telecom companies are against this proposal, arguing it puts them on an unequal 
footing with other internet companies that collect data on users, like Google....”44° In October 27, 2016, 
FCC commissioners in a 3—2 vote delivered a landmark ruling in favor of, in this case, consumer 
protection on the internet. It was a historic day not only in the young life of surveillance capitalism but 
also in the venerable and long life of the FCC, an agency that had never before passed such online 
protections.14! 

Neither the original FCC proposals nor the final vote chilled Verizon’s bid for economies of scale in 
behavioral surplus. If law was coming to its town, it would simply buy a new town without a sheriff. In 
June 2017 Verizon closed on the purchase of Yahoo!’s core business, thus acquiring the former internet 
giant’s one billion active monthly users, including its 600 million monthly active mobile users, for a mere 
$4.48 billion1“ “Scale is imperative,” Armstrong had told journalists a year earlier “If you want to 
play in the Olympics you have to compete against Google and Facebook.” Armstrong touted Verizon’s 
advantages: its complete view of users’ behavior and download activity twenty-four hours per day and its 
continuous tracking of their locations. 

By 2017, the elements of Verizon’s new ambitions were finally in place. The new internet company 
headed by Armstrong and dubbed Oath would combine Yahoo! and AOL for a total of 1.3 billion monthly 
users. As the New York Times summarized, “Verizon hopes to use its range of content and new forms of 
advertising to attract more viewers and marketers as it competes against Google and Facebook.” 

In a chilling epilogue to this chapter in the history of surveillance capitalism, on March 28, 2017, a 
newly elected Republican Congress voted in favor of a resolution to overturn the broadband privacy 
regulations over which the FCC had struggled just months earlier. The rules had required cable and phone 
companies to obtain meaningful consent before using personal information for ads and profiling. The 
companies understood, and they persuaded Republican senators, that the principle of consent would 
strike a serious blow to the foundational mechanisms of the new capitalism: the legitimacy of unilateral 
surplus dispossession, ownership rights to surplus, decision rights over surplus, and the right to lawless 


space for the prosecution of these activities.“° To this end the resolution also prevented the FCC from 
seeking to establish similar protections in the future. Writing in the New York Times, Democratic FCC 
appointee Wheeler went to the heart of the problem: 


To my Democratic colleagues and me, the digital tracks that a consumer leaves when using a network are the property of that 


consumer. They contain private information about personal preferences, health problems and financial matters. Our Republican 


colleagues on the commission argued the data should be available for the network to sel 142 


The reversal meant that although federal laws protected the privacy of a telephone call, the same 
information transmitted by internet immediately enters the ISPs’ surplus supply chains. This roust finally 
signaled the end of the myth of “free.” The Faustian pact that had been sold to the world’s internet users 
posed surveillance as the bitter price of free services such as Google’s Search and Facebook’s social 
network. This obfuscation is no longer tenable, as every consumer who pays his or her monthly telecom 
bill now also purchases the privilege of a remote and abstract but nevertheless rapacious digital strip 
search. 

New and established companies from every sector—including retail, finance, fitness, insurance, 
automotive, travel, hospitality, health, and education—are joining the migratory path to surveillance 
revenues, lured by the magnetism of outsized growth, profit, and the promise of the lavish rewards that 
only the financial markets can confer. We will explore many examples drawn from these sectors in the 
coming chapters. 

In another trend, surveillance in the interest of behavioral surplus capture and sale has become a 
service in its own right. Such companies are often referred to as “software-as-a-service” or SaaS, but 
they are more accurately termed “surveillance as a service,” or “SVaaS.” For example, a new app-based 
approach to lending instantly establishes creditworthiness based on detailed mining of an individual’s 
smartphone and other online behaviors, including texts, e-mails, GPS coordinates, social media posts, 
Facebook profiles, retail transactions, and communication patterns“? Data sources can include intimate 
details such as the frequency with which you charge your phone battery, the number of incoming messages 
you receive, if and when you return phone calls, how many contacts you have listed in your phone, how 
you fill out online forms, or how many miles you travel each day. These behavioral data yield nuanced 
patterns that predict the likelihood of loan default or repayment and thus enable continuous algorithmic 
development and refinement. Two economists who researched this approach discovered that these 
qualities of surplus produce a predictive model comparable to traditional credit scoring, observing that 
“the method quantifies rich aspects of behavior typically considered ‘soft’ information, making it legible 
to formal institutions.”/°° “You’re able to get in and really understand the daily life of these customers,” 
explained the CEO of one lending company that analyzes 10,000 signals per customer.12! 

Such methods were originally developed for markets in Africa to help the “unbanked”—people 
without established credit—to qualify for loans. One lending group interviewed potential customers in 
low-income countries and concluded that it would be easy to exploit the already beleaguered poor: “Most 
said they had no problem sharing personal details in exchange for much-needed funds.” But these app- 
based lending startups are typically developed and funded in Silicon Valley, and it is therefore not 
surprising that the same techniques have become part of a wider trend of exploiting American families 
that have been economically hollowed out by the financial crisis and neoliberalism’s austerity medicine. 
As the Wall Street Journal reports, new startups such as Affirm, LendUp, and ZestFinance “use data from 
sources such as social media, online behavior and data brokers to determine the creditworthiness of tens 
of thousands of U.S. consumers who don’t have access to loans,” more evidence that decision rights and 


the privacy they enable have become luxuries that too many people cannot afford. 

Another example of surveillance-as-a-service is a firm that sells deep vetting of potential employees 
and tenants to employers and landlords. For instance, a prospective tenant receives a demand from her 
potential landlord that requires her to grant full access to all social media profiles. The service then 
“scrapes your site activity,” including entire conversation threads and private messages, runs it through 
natural language processing and other analytic software, and finally spits out a report that catalogues 
everything from your personality to your “financial stress level,” including exposing protected status 
information such as pregnancy and age. There is no opportunity for affected individuals to view or contest 
information. As in the case of digital lenders, although a prospective tenant must formally “opt in” to the 
service, it is those who have less money and fewer options who are trapped in this Faustian bargain in 
which privacy is forfeit to social participation. “People will give up their privacy to get something they 
want,” celebrates the CEO of this service firm.12 

Another genre of SVaaS firms employs data science and machine learning to scour the internet for 
behavioral surplus about individuals, either to sell it or to analyze and fabricate it into lucrative 
prediction products. Legal scholar Frank Pasquale describes this as “the dark market for personal 
data.” 4 For example, hiQ markets its prediction products to corporate human resources professionals. It 
scrapes the web for information related to a client’s employees, including social media and public 
available data; then its “data science engine extracts strong signals from that noise that indicate someone 
may be a flight risk.” Machine learning models assign risk scores to each employee, enabling clients “to 
pinpoint with laser-like accuracy the employees that are highest risk....” The company claims that it 
provides “a crystal ball” and that its predictions are “virtually identical” to observed turnover. With 
hiQ’s information, companies can preemptively intervene. They might make an effort to retain an 
employee, or they may choose to preemptively terminate someone who is predicted to be a “flight 
risk.” 2 

Another example is Safegraph, a company that partners with all those apps that are tracking your 
behavior to amass “high precision/low false positive” data collected “in background from large 
populations.” According to the Washington Post, the company collected 17 trillion location markers from 
10 million smartphones in November 2016 alone, data that were sold to two university researchers, 
among others, for a detailed study of political influences on patterns of family behavior on Thanksgiving 
Day that year.° Despite the widely employed euphemisms of “anonymization” and “deidentification,” 
Safegraph tracks individual devices and the movement of their owners throughout the day, producing data 
that are sufficiently granular to be able to identify individuals’ home locations. 

Surveillance capitalism was born digital, but as we shall see in following chapters, it is no longer 
confined to born-digital companies. This logic for translating investment into revenue is highly adaptive 
and exceptionally lucrative as long as raw-material supplies are free and law is kept at bay. The rapid 
migration to surveillance revenues that is now underway recalls the late-twentieth-century shift from 
revenues derived from goods and services to revenues derived from mastering the speculative and 
shareholder-value-maximizing strategies of financial capitalism. Back then, every company was forced to 
obey the same commandments: shrink head count, offshore manufacturing and service facilities, reduce 
expenditures on product and service quality, diminish commitments to employees and consumers, and 
automate the customer interface, all radical cost-reduction strategies designed to support the firm’s share 
price, which was held hostage to an increasingly narrow and exclusionary view of the firm and its role in 
society. 

As competition for surveillance assets heats up, new laws of motion rise to salience. Eventually, these 
will shape an even-more-merciless imperative to predict future behavior with greater certainty and detail, 


forcing the whole project to break loose from the virtual world in favor of the one that we call “real.” In 
Part II we follow this migration to the real world, as competitive dynamics force the expansion of supply 
operations and an ever-more-complex extraction architecture reaches both further and deeper into new 
territories of human experience. 

Before we undertake that project, it is time to stop and check our bearings. I have suggested that the 
dangers of surveillance capitalism cannot be fully grasped through either the lens of privacy or of 
monopoly. In Chapter 6 I offer a new way of thinking about danger. The threats we face are even more 
fundamental as surveillance capitalists take command of the essential questions that define knowledge, 
authority, and power in our time: Who knows? Who decides? Who decides who decides? 


CHAPTER SIX 


HIUACKED: THE DIVISION OF LEARNING IN 
SOCIETY 


They wondered why the fruit had been forbidden: 

It taught them nothing new. They hid their pride, 

But did not listen much when they were chidden: 
They knew exactly what to do outside. 


—W. H. AUDEN 
SONNETS FROM CHINA, I 


I. The Google Declarations 


On December 4, 1492, Columbus escaped the onshore winds that had prevented his departure from the 
island that we now call Cuba. Within a day he dropped anchor off the coast of a larger island known to its 
people as Quisqueya or Bohio, setting into motion what historians call the “conquest pattern.” It’s a 
design that unfolds in three phases: the invention of legalistic measures to provide the invasion with a 
gloss of justification, a declaration of territorial claims, and the founding of a town to legitimate and 
institutionalize the conquest.t The sailors could not have imagined that their actions that day would write 
the first draft of a pattern whose muscle and genius would echo across space and time to a digital twenty- 
first century. 

On Bohio, Columbus finally found a thriving material culture worthy of his dreams and the appetites of 
the Spanish monarchs. He saw gold, elaborate stone and woodwork, “ceremonial spaces... stone-lined 
ball courts... stone collars, pendants, and stylized statues... richly carved wooden thrones... elaborate 
personal jewelry....” Convinced that the island was “his best find so far, with the most promising 
environment and the most ingenious inhabitants,” he declared to Queen Isabella, “it only remains to 
establish a Spanish presence and order them to perform your will. For... they are yours to command and 
make them work, sow seed, and do whatever else is necessary, and build a town, and teach them to wear 
clothes and adopt our customs.”? 

According to the philosopher of language John Searle, a declaration is a particular way of speaking 
and acting that establishes facts out of thin air, creating a new reality where there was nothing. Here is 
how it works: sometimes we speak to simply describe the world—“you have brown eyes”—or to change 
it—“Shut the door.” A declaration combines both, asserting a new reality by describing the world as if a 


desired change were already true: “All humans are created equal.” “They are yours to command.” As 
Searle writes, “We make something the case by representing it as being the case.”2 

Not every declaration is a spoken statement. Sometimes we just describe, refer to, talk about, think 
about, or even act in relation to a situation in ways that “create a reality by representing that reality as 
created.” For example, let’s say the waiter brings my friend and me two identical bowls of soup, placing 
one bowl in front of each of us. Without saying anything, he has declared that the bowls are not the same: 
one bowl is my friend’s, and the other bowl is mine. We strengthen the facts of his declaration when I take 
soup only from “my” bowl and my friend takes soup from his. When “his” bowl is empty, my friend is 
still hungry, and he asks permission to take a spoonful of soup from the bowl in front of me, further 
establishing the fact that it is my bowl of soup. In this way declarations rise or fall on the strength of 
others’ acceptance of the new facts. As Searle concludes, “All of institutional reality, and therefore... all 
of human civilization is created by... declarations.”4 

Declarations are inherently invasive because they impose new facts on the social world while their 
declarers devise ways to get others to agree to those facts. Columbus’s declaration reflects this “conquest 
pattern,” as historian Matthew Restall writes: 


Sixteenth-century Spaniards consistently presented their deeds and those of their compatriots in terms that prematurely anticipated 
the completion of Conquest campaigns and imbued Conquest chronicles with an air of inevitability. The phrase “Spanish Conquest” 


and all it implies has come down through history because the Spaniards were so concerned to depict their endeavors as conquests 


and pacifications, as contracts fulfilled, as providential intention, as faits accomplis.2 


The Spanish conquerors and their monarchs were eager to justify their invasion as one way to induce 
agreement, especially among their European audience. They developed measures intended to impart “a 
legalistic veneer by citing and following approved precedents.”® To this end the soldiers were tasked 
with reading the Monarchical Edict of 1513 known as the Requirimiento to indigenous villagers before 
attacking them.’ The edict declared that the authority of God, the pope, and the king was embodied in the 
conquistadors and then declared the native peoples as vassals subordinate to that authority: “You Cacics 
and Indians of this Continent.... We declare or be it known to you all, that there is but one God, one hope, 
and one King of Castile, who is Lord of these Countries; appear forth without delay, and take the oath of 
Allegiance to the Spanish King, as his Vassals.”® 

The edict went on to enumerate the sufferings that would befall the villagers if they failed to comply. 
In this world-shattering confrontation with the unprecedented, the native people were summoned, advised, 
and forewarned in a language they could not fathom to surrender without resistance in recognition of 
authorities they could not conceive. The exercise was so cynical and cruel that the approaching invaders 
often dispatched their obligation by mumbling the edict’s long paragraphs into their beards in the dead of 
night as they hid among the thick vegetation waiting to pounce: “Once the Europeans had discharged their 
duty to inform, the way was clear for pillage and enslavement.” The friar Bartolomé de las Casas, whose 
account bears witness to this history of Spanish atrocities, wrote that the Requirimiento promised the 
native people fair treatment upon surrender but also spelled out the consequences of defiance. Every act 
of indigenous resistance was framed as “revolt,” thereby legitimizing brutal “retaliation” that exceeded 
military norms, including grotesque torture, the burning of whole villages in the dark of night, and hanging 
women in public view: “I will do to you all the evil and damages that a lord may do to vassals who do 
not obey or receive him. And I solemnly declare that the deaths and damages received from such will be 
your fault and not that of His Majesty, nor mine, nor of the gentlemen who came with me.”2 

Conquest by declaration should sound familiar because the facts of surveillance capitalism have been 


carried into the world on the strength of six critical declarations pulled from thin air when Google first 
asserted them. That the facts they proclaimed have been allowed to stand is evident in the dispossession 
strategies of Verizon and other new entrants to the surveillance capitalist firmament. In the rapture of the 
young firm’s achievements, Google’s founders, fans, and adoring press passed over in silence the startling 
vision of invasion and conquest concealed in these assertions.12 

The six declarations laid the foundation for the wider project of surveillance capitalism and its 
original sin of dispossession. They must be defended at any cost because each declaration builds on the 


one before it. If one falls, they all fall: 


¢ We claim human experience as raw material free for the taking. On the basis of this claim, we can 
ignore considerations of individuals’ rights, interests, awareness, or comprehension. 

* On the basis of our claim, we assert the right to take an individual’s experience for translation 
into behavioral data. 

* Our right to take, based on our claim of free raw material, confers the right to own the behavioral 
data derived from human experience. 

* Our rights to take and to own confer the right to know what the data disclose. 

* Our rights to take, to own, and to know confer the right to decide how we use our knowledge. 

* Our rights to take, to own, to know, and to decide confer our rights to the conditions that preserve 
our rights to take, to own, to know, and to decide. 


Thus, the age of surveillance capitalism was inaugurated with six declarations that defined it as an age 
of conquest. Surveillance capitalism succeeded by way of aggressive declaration, and its success stands 
as a powerful illustration of the invasive character of declarative words and deeds, which aim to conquer 
by imposing a new reality. These twenty-first-century invaders do not ask permission; they forge ahead, 
papering the scorched earth with faux-legitimation practices. Instead of cynically conveyed monarchical 
edicts, they offer cynically conveyed terms-of-service agreements whose stipulations are just as obscured 
and incomprehensible. They build their fortifications, fiercely defending their claimed territories, while 
gathering strength for the next incursion. Eventually, they build their towns in intricate ecosystems of 
commerce, politics, and culture that declare the legitimacy and inevitability of all that they have 
accomplished. 

Eric Schmidt asked for trust, but Google’s “declarations” ensured that it did not require our trust to 
succeed. Its declarative victories have been the means through which it amassed world-historic 
concentrations of knowledge and power. These are the bulwarks that enable its continued progress. 
Schmidt has occasionally revealed something like this point. When describing “modern technology 
platforms,” he writes that “almost nothing, short of a biological virus, can scale as quickly, efficiently, or 
aggressively as these technology platforms, and this makes the people who build, control, and use them 
powerful too.” # 

On the strength of its unprecedented concentrations of knowledge and power, surveillance capitalism 
achieves dominance over the division of learning in society—the axial principle of social order in an 
information civilization. This development is all the more dangerous because it is unprecedented. It 
cannot be reduced to known harms and therefore does not easily yield to known forms of combat. What is 
this new principle of social order, and how do surveillance capitalists take command of it? These are the 
questions that we pursue in the sections that follow. The answers help us reflect on what we have learned 
and prepare for what lies ahead. 


IT. Who Knows? 


This book began by recalling an urgent question posed to me by a young pulp mill manager in a small 
southern town: “Are we all going to be working for a smart machine, or will we have smart people 
around the machine?” In the years that followed that rainy evening, I closely observed the digitalization 
of work in the pulp mill. As I described it in In the Age of the Smart Machine, the shift to information 
technology transformed the mill into an “electronic text” that became the primary focus of every worker’s 
attention. Instead of the hands-on tasks associated with raw materials and equipment, doing “a good job” 
came tO Mean monitoring data on screens and mastering the skills to understand, learn from, and act 
through the medium of this electronic text. What seems ordinary today was extraordinary then. 

These obvious changes, I argued, signaled a deep and significant transformation. The ordering 
principle of the workplace had shifted from a division of labor to a division of learning. I wrote about 
the many women and men who surprised themselves and their managers as they conquered new 
intellectual skills and learned to thrive in the information-rich environment, but I also documented the 
bitter conflicts that attended those achievements, summarized as dilemmas of knowledge, authority, and 
power. 

Any consideration of the division of learning must resolve these dilemmas expressed in three essential 
questions. The first question is “Who knows?” This is a question about the distribution of knowledge and 
whether one is included or excluded from the opportunity to learn. The second question is “Who 
decides?” This is a question about authority: which people, institutions, or processes determine who is 
included in learning, what they are able to learn, and how they are able to act on their knowledge. What is 
the legitimate basis of that authority? The third question is “Who decides who decides?” This is a 
question about power. What is the source of power that undergirds the authority to share or withhold 
knowledge? 

The young manager would ultimately find his answers, but they were not what either of us had hoped 
for. Even as the pulp mill workers struggled and often triumphed, Hayek’s worldview was taking hold at 
the highest policy levels and Jensen’s operational disciplines were finding an eager welcome on Wall 
Street, which quickly learned to impose them on every public company. The result was a cost-down 
business model oriented to its Wall Street audience, which insisted on automating and exporting jobs 
rather than investing in the digital skills and capabilities of the US worker. The answer to the question 
Who knows? was that the machine knows, along with an elite cadre able to wield the analytic tools to 
troubleshoot and extract value from information. The answer to Who decides? was a narrow market form 
and its business models that decide. Finally, in the absence of a meaningful double movement, the answer 
to Who decides who decides? defaults entirely to financial capital bound to the disciplines of 
shareholder-value maximization. 

It is not surprising that nearly forty years later, a Brookings Institution report laments that millions of 
US workers are “shut out of decent middle-skill opportunities” in the face of “rapid digitalization.” The 
report exhorts companies to “invest urgently in IT upskilling strategies for incumbent workers, knowing 
that digital skills represent a key channel of productivity gains.”42 How different might our society be if 
US businesses had chosen to invest in people as well as in machines? 

Most companies opted for the smart machine over smart people, producing a well-documented pattern 
that favors substituting machines and their algorithms for human contributors in a wide range of jobs. By 
now, these include many occupations far from the factory floor? This results in what economists call 


“job polarization,” which features some high-skill jobs and other low-skill jobs, with automation 
displacing most of the jobs that were once “in the middle.”44 And although some business leaders, 
economists, and technologists describe these developments as necessary and inevitable consequences of 
computer-based technologies, research shows that the division of learning in the economic domain 
reflects the strength of neoliberal ideology, politics, culture, and institutional patterns. For example, in 
continental and northern Europe, where key elements of the double movement have survived in some 
form, job polarization is moderated by substantial investments in workforce education that produce a 
more inclusive division of learning as well as high-quality innovative products and services. 

Most critical to our story is that we now face a second historical phase of this conflict. The division of 
learning in the economic domain of production and employment is critical, but it is only the beginning of a 
new struggle over the even larger question of the division of learning in society. The dilemmas of 
knowledge, authority, and power have burst through the walls of the workplace to overwhelm our daily 
lives. As people, processes, and things are reinvented as information, the division of learning in society 
becomes the ascendant principle of social ordering in our time. 

A wholly new electronic text now extends far beyond the confines of the factory or office. Thanks to 
our computers, credit cards, and phones, and the cameras and sensors that proliferate in public and 
private spaces, just about everything we now do is mediated by computers that record and codify the 
details of our daily lives at a scale that would have been unimaginable only a few years ago. We have 
reached the point at which there is little that is omitted from the continuous accretion of this new 
electronic text. In later chapters we review many illustrations of the new electronic text as it spreads 
silently but relentlessly, like a colossal oil slick engulfing everything in its path: your breakfast 
conversation, the streets in your neighborhood, the dimensions of your living room, your run in the park. 

The result is that both the world and our lives are pervasively rendered as information. Whether you 
are complaining about your acne or engaging in political debate on Facebook, searching for a recipe or 
sensitive health information on Google, ordering laundry soap or taking photos of your nine-year-old, 
smiling or thinking angry thoughts, watching TV or doing wheelies in the parking lot, all of it is raw 
material for this burgeoning text. Information scholar Martin Hilbert and his colleagues observe that even 
the foundational elements of civilization, including “language, cultural assets, traditions, institutions, 
rules, and laws... are currently being digitized, and for the first time, explicitly put into visible code,” 
then returned to society through the filter of “intelligent algorithms” deployed to govern a rapidly 
multiplying range of commercial, governmental, and social functions.!° The essential questions confront 
us at every turn: Who knows? Who decides? Who decides who decides? 


III. Surveillance Capital and the Two Texts 


There are important parallels with the late nineteenth and early twentieth centuries, when the division of 
labor first emerged as the foremost principle of social organization in the nascent industrial societies of 
Europe and North America. These experiences can offer guidance and alert us to what is at stake. For 
example, when the young Emile Durkheim wrote The Division of Labor in Society, the title itself was 
controversial. The division of labor had been understood as a critical means of achieving labor 
productivity through the specialization of tasks. Adam Smith memorably wrote about this new principle of 
industrial organization in his description of a pin factory, and the division of labor remained a topic of 
economic discourse and controversy throughout the nineteenth century. Durkheim recognized labor 
productivity as an economic imperative of industrial capitalism that would drive the division of labor to 


its most extreme application, but that was not what held his fascination. 

Instead, Durkheim trained his sights on the social transformation already gathering around him, 
observing that “specialization” was gaining “influence” in politics, administration, the judiciary, science, 
and the arts. He concluded that the division of labor was no longer quarantined in the industrial 
workplace. Instead, it had burst through those factory walls to becoming the critical organizing principle 
of industrial society. This is also an example of Edison’s insight: that the principles of capitalism initially 
aimed at production eventually shape the wider social and moral milieu. “Whatever opinion one has 
about the division of labor,” Durkheim wrote, “everyone knows that it exists, and is more and more 
becoming one of the fundamental bases of the social order.” “ 

Economic imperatives predictably mandated the division of labor in production, but what was the 
purpose of the division of labor in society? This was the question that motivated Durkheim’s analysis, and 
his century-old conclusions are still relevant for us now. He argued that the division of labor accounts for 
the interdependencies and reciprocities that link the many diverse members of a modern industrial society 
in a larger prospect of solidarity. Reciprocities breed mutual need, engagement, and respect, all of which 
imbue this new ordering principle with moral force. 

In other words, the division of labor was summoned into society at the beginning of the twentieth 
century by the rapidly changing circumstances of the first modernity’s new individuals, discussed in 
Chapter 2. It was an essential response to their new “conditions of existence.” As people like my great- 
grandparents joined the migration to a modern world, the old sources of meaning that had bonded 
communities across space and time melted away. What would hold society together in the absence of the 
rules and rituals of clan and kin? Durkheim’s answer was the division of labor. People’s needs for a 
coherent new source of meaning and structure were the cause, and the effect was an ordering principle 
that enabled and sustained a healthy modern community. As the young sociologist explained, 


The most remarkable effect of the division of labor is not that it increases output of functions divided, but that it renders them solidary. 
Its role... is not simply to embellish or ameliorate existing societies, but to render societies possible which, without it, would not exist. 


... It passes far beyond purely economic interests, for it consists in the establishment of a social and moral order sui generis.18 


Durkheim’s vision was neither sterile nor naive. He recognized that things can take a dark turn and 
often do, resulting in what he called an “abnormal” (sometimes translated as “pathological”) division of 
labor that produces social distance, injustice, and discord in place of reciprocity and interdependency. In 
this context, Durkheim singled out the destructive effects of social inequality on the division of labor in 
society, especially what he viewed as the most dangerous form of inequality: extreme asymmetries of 
power that make “conflict itself impossible” by “refusing to admit the right of combat.” Such pathologies 
can be cured only by a politics that asserts the people’s right to contest, confront, and prevail in the face 
of unequal and illegitimate power over society. In the late nineteenth century and most of the twentieth 
century, that contest was led by labor and other social movements that asserted social equality through 
institutions such as collective bargaining and public education. 

The transformation that we witness in our time echoes these historical observations as the division of 
learning follows the same migratory path from the economic to the social domain once traveled by the 
division of labor. Now the division of learning “passes far beyond purely economic interests,” for it 
establishes the basis for our social order and its moral content. 

The division of learning is to us, members of the second modernity, what the division of labor was to 
our grandparents and great-grandparents, pioneers of the first modernity. In our time the division of 
learning emerges from the economic sphere as a new principle of social order and reflects the primacy of 


learning, information, and knowledge in today’s quest for effective life. And just as Durkheim warned his 
society a century ago, today our societies are threatened as the division of learning drifts into pathology 
and injustice at the hands of the unprecedented asymmetries of knowledge and power that surveillance 
capitalism has achieved. 

Surveillance capitalism’s command of the division of learning in society begins with what I call the 
problem of the two texts. The specific mechanisms of surveillance capitalism compel the production of 
two “electronic texts,” not just one. When it comes to the first text, we are its authors and readers. This 
public-facing text is familiar and celebrated for the universe of information and connection it brings to our 
fingertips. Google Search codifies the informational content of the world wide web. Facebook’s News 
Feed binds the network. Much of this public-facing text is composed of what we inscribe on its pages: our 
posts, blogs, videos, photos, conversations, music, stories, observations, “likes,” tweets, and all the great 
massing hubbub of our lives captured and communicated. 

Under the regime of surveillance capitalism, however, the first text does not stand alone; it trails a 
shadow close behind. The first text, full of promise, actually functions as the supply operation for the 
second text: the shadow text. Everything that we contribute to the first text, no matter how trivial or 
fleeting, becomes a target for surplus extraction. That surplus fills the pages of the second text. This one is 
hidden from our view: “read only” for surveillance capitalists.22 In this text our experience is dragooned 
as raw material to be accumulated and analyzed as means to others’ market ends. The shadow text is a 
burgeoning accumulation of behavioral surplus and its analyses, and it says more about us than we can 
know about ourselves. Worse still, it becomes increasingly difficult, and perhaps impossible, to refrain 
from contributing to the shadow text. It automatically feeds on our experience as we engage in the normal 
and necessary routines of social participation. 

More mystifying still are the ways in which surveillance capitalists apply what they learn from their 
exclusive shadow text to shape the public text to their interests. There have been myriad revelations of 
Google and Facebook’s manipulations of the information that we see. For now I'll simply point out that 
Google’s algorithms, derived from surplus, select and order search results, and Facebook’s algorithms, 
derived from surplus, select and order the content of its News Feed. In both cases, researchers have 
shown that these manipulations reflect each corporation’s commercial objectives. As legal scholar Frank 
Pasquale describes it, “The decisions at the Googleplex are made behind closed doors... the power to 
include, exclude, and rank is the power to ensure which public impressions become permanent and which 
remain fleeting.... Despite their claims of objectivity and neutrality, they are constantly making value- 
laden, controversial decisions. They help create the world they claim to merely ‘show’ us.”22 When it 
comes to the shadow text, surveillance capitalism’s laws of motion compel both its secrecy and its 
continuous growth. We are the objects of its narratives, from whose lessons we are excluded. As the 
source from which all the treasure flows, this second text is about us, but it is not for us. Instead, it is 
created, maintained, and exploited outside our awareness for others’ benefit. 

The result is that the division of learning is both the ascendant principle of social ordering in our 
information civilization and already a hostage to surveillance capitalism’s privileged position as the 
dominant composer, owner, and guardian of the texts. Surveillance capitalism’s ability to corrupt and 
control these texts produces unprecedented asymmetries of knowledge and power that operate precisely 
as Durkheim had feared: the relatively free rein accorded to this market form and the innately illegible 
character of its action have enabled it to impose substantial control over the division of learning outside 
of our awareness and without means of combat. When it comes to the essential questions, surveillance 
capital has gathered the power and asserted the authority to supply all the answers. However, even 
authority is not enough. Only surveillance capital commands the material infrastructure and expert 


brainpower to rule the division of learning in society. 


IV. The New Priesthood 


Scientists warn that the world’s capacity to produce information has substantially exceeded its ability to 
process and store information. Consider that our technological memory has roughly doubled about every 
three years. In 1986 only 1 percent of the world’s information was digitized and 25 percent in 2000. By 
2013, the progress of digitalization and datafication (the application of software that allows computers 
and algorithms to process and analyze raw data) combined with new and cheaper storage technologies 
had translated 98 percent of the world’s information into a digital format.2! 

Information is digital, but its volume exceeds our ability to discern its meaning. As the solution to this 
problem, information scholar Martin Hilbert counsels, “The only option we have left to make sense of all 
the data is to fight fire with fire,” using “artificially intelligent computers” to “sift through the vast 
amounts of information.... Facebook, Amazon, and Google have promised to... create value out of vast 
amounts of data through intelligent computational analysis.”2* The rise of surveillance capitalism 
necessarily turns Hilbert’s advice into a dangerous proposition. Although he does not mean to, Hilbert 
merely confirms the privileged position of the surveillance capitalists and the asymmetrical power that 
enables them to bend the division of learning to their interests. 

Google’s asymmetrical power draws on all the social sources that we have considered: its 
declarations, its defensive fortifications, its exploitation of law, the legacy of surveillance 
exceptionalism, the burdens of second-modernity individuals, and so on. But its power would not be 
operational without the gargantuan material infrastructure that surveillance revenues have bought. Google 
is the pioneer of “hyperscale,” considered to be “the largest computer network on Earth.”22 Hyperscale 
operations are found in high-volume information businesses such as telecoms and global payments firms, 
where data centers require millions of “virtual servers” that exponentially increase computing 
capabilities without requiring substantial expansion of physical space, cooling, or electrical power 
demands.24 The machine intelligence at the heart of Google’s formidable dominance is described as “80 
percent infrastructure,” a system that comprises custom-built, warehouse-sized data centers spanning 15 
locations and, in 2016, an estimated 2.5 million servers in four continents.22 

Investors deem Google “harder to catch than ever” because it is unmatched in its combination of 
infrastructure scale and science. Google is known as a “full stack AI company” that uses its own data 
stores “to train its own algorithms running on its own chips deployed on its own cloud.” Its dominance is 
further strengthened by the fact that machine learning is only as intelligent as the amount of data it has to 
train on, and Google has the most data.2° By 2013, the company understood that its shift into the “neural 
networks” that define the current frontier of artificial intelligence would substantially increase 
computational demands and require a doubling of its data centers. As Urs Hdélzle, Google’s senior vice 
president of technical infrastructure, put it, “The dirty secret behind [AI] is that they require an insane 
number of computations to just actually train the network.” If the company had tried to process the 
growing computational workload with traditional CPUs, he explained, “We would have had to double the 
entire footprint of Google—data centers and servers—just to do three minutes or two minutes of speech 
recognition per Android user per day.”Z 

With data center construction as the company’s largest line item and power as its highest operating 
cost, Google invented its way through the infrastructure crisis. In 2016 it announced the development of a 
new chip for “deep learning inference” called the tensor processing unit (TPU). The TPU would 


dramatically expand Google’s machine intelligence capabilities, consume only a fraction of the power 
required by existing processors, and reduce both capital expenditure and the operational budget, all while 
learning more and faster.8 

Global revenue for AI products and services is expected to increase 56-fold, from $644 million in 
2016 to $36 billion in 2025.22 The science required to exploit this vast opportunity and the material 
infrastructure that makes it possible have ignited an arms race among tech companies for the 10,000 or so 
professionals on the planet who know how to wield the technologies of machine intelligence to coax 
knowledge from an otherwise cacophonous data continent. Google/Alphabet is the most aggressive 
acquirer of AI technology and talent. In 2014—2016 it purchased nine AI companies, twice as many as its 
nearest rival, Apple. 

The concentration of AI talent at Google reflects a larger trend. In 2017, US companies are estimated 
to have allocated more than $650 million to fuel the AI talent race, with more than 10,000 available 
positions at top employers across the country. The top five tech companies have the capital to crowd out 
competitors: startups, universities, municipalities, established corporations in other industries, and less 
wealthy countries.2! In Britain, university administrators are already talking about a “missing generation” 
of data scientists. The huge salaries of the tech firms have lured so many professionals that there is no one 
left to teach the next generation of students. As one scholar described it, “The real problem is these 
people are not dispersed through society. The intellect and expertise is concentrated in a small number of 
companies.”=2 

On the strength of its lavish recruitment efforts, Google tripled its number of machine intelligence 
scientists in just the last few years and has become the top contributor to the most prestigious scientific 
journals—four to five times the world average in 2016. Under the regime of surveillance capitalism, the 
corporation’s scientists are not recruited to solve world hunger or eliminate carbon-based fuels. Instead, 
their genius is meant to storm the gates of human experience, transforming it into data and translating it 
into a new market colossus that creates wealth by predicting, influencing, and controlling human behavior. 

More than six hundred years ago, the printing press put the written word into the hands of ordinary 
people, rescuing the prayers, bypassing the priesthood, and delivering the opportunity for spiritual 
communion directly into the hands of the prayerful. We have come to take for granted that the internet 
enables an unparalleled diffusion of information, promising more knowledge for more people: a mighty 
democratizing force that exponentially realizes Gutenberg’s revolution in the lives of billions of 
individuals. But this grand achievement has blinded us to a different historical development, one that 
moves out of range and out of sight, designed to exclude, confuse, and obscure. In this hidden movement 
the competitive struggle over surveillance revenues reverts to the pre-Gutenberg order as the division of 
learning in society shades toward the pathological, captured by a narrow priesthood of privately 
employed computational specialists, their privately owned machines, and the economic interests for 
whose sake they learn. 


V. The Privatization of the Division of Learning in Society 


The division of learning in society has been hijacked by surveillance capitalism. In the absence of a 
robust double movement in which democratic institutions and civil society tether raw information 
capitalism to the people’s interests—however imperfectly—we are thrown back on the market form of the 
surveillance capitalist companies in this most decisive of contests over the division of learning in society. 
Experts in the disciplines associated with machine intelligence know this, but they have little grasp of its 


wider implications. As data scientist Pedro Domingos writes, “Whoever has the best algorithms and the 
most data wins.... Google with its head start and larger market share, knows better what you want... 
whoever learns fastest wins....” The New York Times reports that Google CEO Sundar Pichai now shares 
a floor with the company’s AI research lab and notes it as a trend among many CEOs: a literal take on the 
concentration of power.22 

Just over thirty years ago, legal scholar Spiros Simitis published a seminal essay on the theme of 
privacy in an information society. Simitis grasped early on that the already visible trends in public and 
private “information processing” harbored threats to society that transcended narrow conceptions of 
privacy and data ownership: “Personal information is increasingly used to enforce standards of behavior. 
Information processing is developing, therefore, into an essential element of long-term strategies of 
manipulation intended to mold and adjust individual conduct.”“4 Simitis argued that these trends were 
incompatible not only with privacy but with the very possibility of democracy, which depends upon a 
reservoir of individual capabilities associated with autonomous moral judgment and self-determination. 

Building on Simitis’s work, Berkeley’s Paul M. Schwartz warned in 1989 that computerization would 
transform the delicate balance of rights and obligations upon which privacy law depends: “Today the 
enormous amounts of personal data available in computers threaten the individual in a way that renders 
obsolete much of the previous legal protection.” Most important, Schwartz foresaw that the scale of the 
still-emerging crisis would impose risks that exceed the scope of privacy law: “The danger that the 
computer poses is to human autonomy. The more that is known about a person, the easier it is to control 
him. Insuring the liberty that nourishes democracy requires a structuring of societal use of information and 
even permitting some concealment of information.”=2 

Both Simitis and Schwartz sensed the ascent of the division of learning as the axial principle of a new 
computational societal milieu, but they could not have anticipated the rise of surveillance capitalism and 
its consequences. Although the explosive growth of the information continent shifts a crucial axis of the 
social order from a twentieth-century division of labor to a twenty-first-century division of learning, it is 
surveillance capitalists who command the field and unilaterally lay claim to a disproportionate share of 
the decision rights that shape the division of learning in society. 

Surveillance capitalists’ acts of digital dispossession impose a new kind of control upon individuals, 
populations, and whole societies. Individual privacy is a casualty of this control, and its defense requires 
a reframing of privacy discourse, law, and judicial reasoning. The “invasion of privacy’ is now a 
predictable dimension of social inequality, but it does not stand alone. It is the systematic result of a 
“pathological” division of learning in society in which surveillance capitalism knows, decides, and 
decides who decides. Demanding privacy from surveillance capitalists or lobbying for an end to 
commercial surveillance on the internet is like asking Henry Ford to make each Model T by hand or 
asking a giraffe to shorten its neck. Such demands are existential threats. They violate the basic 
mechanisms and laws of motion that produce this market leviathan’s concentrations of knowledge, power, 
and wealth. 

So here is what is at stake: surveillance capitalism is profoundly antidemocratic, but its remarkable 
power does not originate in the state, as has historically been the case. Its effects cannot be reduced to or 
explained by technology or the bad intentions of bad people; they are the consistent and predictable 
consequences of an internally consistent and successful logic of accumulation. Surveillance capitalism 
rose to dominance in the US under conditions of relative lawlessness. From there it spread to Europe, and 
it continues to make inroads in every region of the world. Surveillance capitalist firms, beginning with 
Google, dominate the accumulation and processing of information, especially information about human 
behavior. They know a great deal about us, but our access to their knowledge is sparse: hidden in the 


shadow text and read only by the new priests, their bosses, and their machines. 

This unprecedented concentration of knowledge produces an equally unprecedented concentration of 
power: asymmetries that must be understood as the unauthorized privatization of the division of 
learning in society. This means that powerful private interests are in control of the definitive principle of 
social ordering in our time, just as Durkheim warned of the subversion of the division of labor by the 
powerful forces of industrial capital a century ago. As things currently stand, it is the surveillance 
capitalist corporations that know. It is the market form that decides. It is the competitive struggle among 
surveillance capitalists that decides who decides. 


VI. The Power of the Unprecedented: A Review 


The titanic power struggles of the twentieth century were between industrial capital and labor, but the 
twenty-first century finds surveillance capital pitted against the entirety of our societies, right down to 
each individual member. The competition for surveillance revenues bears down on our bodies, our 
homes, and our cities in a battle for power and profit as violent as any the world has seen. Surveillance 
capitalism cannot be imagined as something “out there” in factories and offices. Its aims and effects are 
here... are us. 

Ours is not simply a case of being ambushed and outgunned. We were caught off guard because there 
was no way that we could have imagined these acts of invasion and dispossession, any more than the first 
unsuspecting Taino cacique could have foreseen the rivers of blood that would flow from his inaugural 
gesture of hospitality toward the hairy, grunting, sweating men, the adelantados who appeared out of thin 
air waving the banner of the Spanish monarchs and their pope as they trudged across the beach. Why have 
we been slow to recognize the “original sin of simple robbery” at the heart of this new capitalism? Like 
the Tainos, we faced something altogether new to our story: the unprecedented. And, like them, we risk 
catastrophe when we assess new threats through the lens of old experience. 

On the “supply side,” surveillance capitalists deftly employed the entire arsenal of the declaration to 
assert their authority and legitimacy in a new and undefended digital world. They used declarations to 
take without asking. They camouflaged their purpose with illegible machine operations, moved at extreme 
velocities, sheltered secretive corporate practices, mastered rhetorical misdirection, taught helplessness, 
purposefully misappropriated cultural signs and symbols associated with the themes of the second 
modernity—empowerment, participation, voice, individualization, collaboration—and baldly appealed to 
the frustrations of second-modernity individuals thwarted in the collision between psychological yearning 
and institutional indifference. 

In this process the pioneer surveillance capitalists at Google and Facebook evaded the disciplines of 
corporate governance and rejected the disciplines of democracy, protecting their claims with financial 
influence and political relationships. Finally, they benefited from history, born in a time when regulation 
was equated with tyranny and the state of exception precipitated by the terrorist attacks of 9/11 produced 
surveillance exceptionalism, further enabling the new market to root and flourish. Surveillance 
capitalists’ purposeful strategies and accidental gifts produced a form that can romance and beguile but is 
also ruthlessly efficient at extinguishing space for democratic deliberation, social debate, individual self- 
determination, and the right to combat as it forecloses every path to exit. 

On the “demand side,” second-modernity populations starved for enabling resources were so 
enraptured by the plentiful bags of rice and powdered milk thrown from the back of the digital truck that 
little attention was paid to the drivers or their destination. We needed them; we even believed that we 


couldn’t live without them. But under scrutiny, those long-awaited delivery trucks look more like 
automated vehicles of invasion and conquest: more Mad Max than Red Cross, more Black Sails than 
Carnival Cruise. The wizards behind their steering wheels careen across every hill and hollow, learning 
how to scrape and stockpile our behavior over which they unabashedly assert their rights as conquerors’ 
plunder. 

In the absence of a clear-minded appreciation of this new logic of accumulation, every attempt at 
understanding, predicting, regulating, or prohibiting the activities of surveillance capitalists will fall 
short. The primary frameworks through which our societies have sought to assert control over 
surveillance capitalism’s audacity are those of “privacy rights” and “monopoly.” Neither the pursuit of 
privacy regulations nor the imposition of constraints on traditional monopoly practices has so far 
interrupted the key mechanisms of accumulation, from supply routes to behavioral futures markets. On the 
contrary, surveillance capitalists have extended and elaborated their extraction architectures across every 
human domain as they master the practical and political requirements of the dispossession cycle. This 
success now threatens the deepest principles of social order in an information civilization as surveillance 
capitalism takes unauthorized command over the division of learning in society. 

If there is to be a fight, let it be a fight over capitalism. Let it be an insistence that raw surveillance 
capitalism is as much a threat to society as it is to capitalism itself. This is not a technical undertaking, not 
a program for advanced encryption, improved data anonymity, or data ownership. Such strategies only 
acknowledge the inevitability of commercial surveillance. They leave us hiding in our own lives as we 
cede control to those who feast on our behavior for their own purposes. Surveillance capitalism depends 
on the social, and it is only in and through collective social action that the larger promise of an 
information capitalism aligned with a flourishing third modernity can be reclaimed. 

In Part I we have seen how Google built its extraction architecture in the online world. As competition 
for surveillance revenues intensified, a second economic imperative rose to prominence driving an 
expansion of that architecture into another world, the one that we call “real.” 

Now the story of surveillance capitalism moves in this new direction. In Part II, I invite you to 
rekindle your sense of astonishment as we follow the trail of this second economic imperative defined by 
the prediction of human behavior. The prediction imperative enlarges the complexity of surplus 
operations as economies of scale are joined by economies of scope and economies of action. These new 
disciplines drive surveillance capitalism far into the intimate reaches of our daily lives and deep into our 
personalities and our emotions. Ultimately, they compel the development of highly inventive but resolutely 
secret new means to interrupt and modify our behavior for the sake of surveillance revenues. These 
operations challenge our elemental right to the future tense, which is the right to act free of the influence 
of illegitimate forces that operate outside our awareness to influence, modify, and condition our behavior. 
We grow numb to these incursions and the ways in which they deform our lives. We succumb to the 
drumbeat of inevitability, but nothing here is inevitable. Astonishment is lost but can be found again. 


CHAPTER SEVEN 


THE REALITY BUSINESS 


Falling in love with Truth before he knew Her, 
He rode into imaginary lands, 
By solitude and fasting hoped to woo Her, 
And mocked at those who served Her with their hands. 


—W. H. AUDEN 
SONNETS FROM CHINA, VI 


I, The Prediction Imperative 


There could not have been a more fitting setting for Eric Schmidt to share his opinion on the future of the 
web than the World Economic Forum in Davos, Switzerland. In 2015, during a session at the winter 
playground for neoliberals—and increasingly surveillance capitalists—Schmidt was asked for his 
thoughts about the future of the internet. Sitting alongside his former Google colleagues Sheryl Sandberg 
and Marissa Mayer, he did not hesitate to share his belief that “The internet will disappear. There will be 
so many IP addresses... so many devices, sensors, things that you are wearing, things that you are 
interacting with, that you won’t even sense it. It will be part of your presence all the time. Imagine you 
walk into a room and the room is dynamic.”! The audience gasped in astonishment, and shortly thereafter, 
headlines around the world exploded in shock at the former Google CEO’s pronouncement that the end of 
the internet was at hand. 

Schmidt was, in fact, merely paraphrasing computer scientist Mark Weiser’s seminal 1991 article, 
“The Computer for the 21st Century,” which has framed Silicon Valley’s technology objectives for nearly 
three decades. Weiser introduced what he called “ubiquitous computing” with two legendary sentences: 
“The most profound technologies are those that disappear. They weave themselves into the fabric of 
everyday life until they are indistinguishable from it.” He described a new way of thinking “that allows 
the computers themselves to vanish into the background.... Machines that fit the human environment 
instead of forcing humans to enter theirs will make using a computer as refreshing as taking a walk in the 
woods.”2 

Weiser understood that the virtual world could never be more than a shadow land no matter how much 
data it absorbs: “Virtual reality is only a map, not a territory. It excludes desks, offices, other people... 
weather, trees, walks, chance encounters and, in general, the infinite richness of the universe.” He wrote 


that virtual reality “simulates” the world rather than “invisibly enhancing the world that already exists.” In 
contrast, ubiquitous computing would infuse that real world with a universally networked apparatus of 
silent, “calm,” and voracious computing. Weiser refers to this apparatus as the new “computing 
environment” and delights in the possibilities of its limitless knowledge, such as knowing “the suit you 
looked at for a long time last week because it knows both of your locations, and it can retroactively find 
the designer’s name even though that information did not interest you at the time.”2 

Schmidt was not describing the end of the internet but rather its successful unshackling from dedicated 
devices such as the personal computer and the smartphone. For surveillance capitalists, this transition is 
not a choice. Surveillance profits awakened intense competition over the revenues that flow from new 
markets for future behavior. Even the most sophisticated process of converting behavioral surplus into 
products that accurately forecast the future is only as good as the raw material available for processing. 
Surveillance capitalists therefore must ask this: what forms of surplus enable the fabrication of prediction 
products that most reliably foretell the future? This question marks a critical turning point in the trial-and- 
error elaboration of surveillance capitalism. It crystallizes a second economic imperative—the 
prediction imperative—and reveals the intense pressure that it exerts on surveillance capitalist revenues. 

The first wave of prediction products enabled targeted online advertising. These products depended 
upon surplus derived at scale from the internet. I have summarized the competitive forces that drive the 
need for surplus at scale as the “extraction imperative.” Competition for surveillance revenues eventually 
reached a point at which the volume of surplus became a necessary but insufficient condition for success. 
The next threshold was defined by the quality of prediction products. In the race for higher degrees of 
certainty, it became clear that the best predictions would have to approximate observation. The prediction 
imperative is the expression of these competitive forces. (See Figure 3 here.) 

Google/Alphabet, Facebook, Microsoft, and many more companies now drawn to surveillance 
revenues have staked their claims on the internet’s “disappearance” because they must. Compelled to 
improve predictions, surveillance capitalists such as Google understood that they had to widen and 
diversify their extraction architectures to accommodate new sources of surplus and new supply 
operations. Economies of scale would still be vital, of course, but in this new phase, supply operations 
were enlarged and intensified to accommodate economies of scope and economies of action. What does 
this entail? 

The shift toward economies of scope defines a new set of aims: behavioral surplus must be vast, but it 
must also be varied. These variations are developed along two dimensions. The first is the extension of 
extraction operations from the virtual world into the “real” world, where we actually live our actual 
lives. Surveillance capitalists understood that their future wealth would depend upon new supply routes 
that extend to real life on the roads, among the trees, throughout the cities. Extension wants your 
bloodstream and your bed, your breakfast conversation, your commute, your run, your refrigerator, your 
parking space, your living room. 

Economies of scope also proceed along a second dimension: depth. The drive for economies of scope 
in the depth dimension is even more audacious. The idea here is that highly predictive, and therefore 
highly lucrative, behavioral surplus would be plumbed from intimate patterns of the self. These supply 
operations are aimed at your personality, moods, and emotions, your lies and vulnerabilities. Every level 
of intimacy would have to be automatically captured and flattened into a tidal flow of data points for the 
factory conveyor belts that proceed toward manufactured certainty. 

Just as scale became necessary but insufficient for higher-quality predictions, it was also clear that 
economies of scope would be necessary but insufficient for the highest quality of prediction products able 
to sustain competitive advantage in the new markets for future behavior. Behavioral surplus must be vast 


and varied, but the surest way to predict behavior is to intervene at its source and shape it. The processes 
invented to achieve this goal are what I call economies of action. In order to achieve these economies, 
machine processes are configured to intervene in the state of play in the real world among real people and 
things. These interventions are designed to enhance certainty by doing things: they nudge, tune, herd, 
manipulate, and modify behavior in specific directions by executing actions as subtle as inserting a 
specific phrase into your Facebook news feed, timing the appearance of a BUY button on your phone, or 
shutting down your car engine when an insurance payment is late. 

This new level of competitive intensity characterized by scope and action ratchets up the invasive 
character of supply operations and initiates a new era of surveillance commerce that I call the reality 
business. Economies of scale were implemented by machine-based extraction architectures in the online 
world. Now the reality business requires machine-based architectures in the real world. These finally 
fulfill Weiser’s vision of ubiquitous automated computational processes that “weave themselves into the 
fabric of everyday life until they are indistinguishable from it,” but with a twist. Now they operate in the 
interests of surveillance capitalists. 

There are many buzzwords that gloss over these operations and their economic origins: “ambient 
computing,” “ubiquitous computing,” and the “internet of things” are but a few examples. For now I will 
refer to this whole complex more generally as the “apparatus.” Although the labels differ, they share a 
consistent vision: the everywhere, always-on instrumentation, datafication, connection, communication, 
and computation of all things, animate and inanimate, and all processes—natural, human, physiological, 
chemical, machine, administrative, vehicular, financial. Real-world activity is continuously rendered 
from phones, cars, streets, homes, shops, bodies, trees, buildings, airports, and cities back to the digital 
realm, where it finds new life as data ready for transformation into predictions, all of it filling the ever- 
expanding pages of the shadow text.4 

As the prediction imperative gathers force, it gradually becomes clear that extraction was the first 
phase of a far-more-ambitious project. Economies of action mean that real-world machine architectures 
must be able to know as well as to do. Extraction is not enough; now it must be twinned with execution. 
The extraction architecture is combined with a new execution architecture, through which hidden 
economic objectives are imposed upon the vast and varied field of behavior. 

Gradually, as surveillance capitalism’s imperatives and the material infrastructures that perform 
extraction-and-execution operations begin to function as a coherent whole, they produce a twenty-first- 
century “means of behavioral modification.” The aim of this undertaking is not to impose behavioral 
norms, such as conformity or obedience, but rather to produce behavior that reliably, definitively, and 
certainly leads to desired commercial results. The research director of Gartner, the well-respected 
business advisory and research firm, makes the point unambiguously when he observes that mastery of the 
“internet of things” will serve as “a key enabler in the transformation of business models from 
‘guaranteed levels of performance’ to ‘guaranteed outcomes.’”® 

This is an extraordinary statement because there can be no such guarantees in the absence of the power 
to make it so. This wider complex that we refer to as the “means of behavioral modification” is the 
expression of this gathering power. The prospect of guaranteed outcomes alerts us to the force of the 
prediction imperative, which demands that surveillance capitalists make the future for the sake of 
predicting it. Under this regime, ubiquitous computing is not just a knowing machine; it is an actuating 
machine designed to produce more certainty about us and for them. 

This gradually accruing, smart, and muscular apparatus is gradually being assembled around us. No 
one knows what the real magnitude is or will be. It is a domain plagued by hyperbole, where projections 
frequently outrun actual results. Despite this, the planning, investment, and invention necessary to draw 


this vision of ubiquity into reality are well underway. The visions and aims of its architects, the work that 
has already been accomplished, and the programs that are currently in development constitute a turning 
point in the evolution of surveillance capitalism. 

Finally, I want to underscore that although it may be possible to imagine something like the “internet of 
things” without surveillance capitalism, it is impossible to imagine surveillance capitalism without 
something like the “internet of things.” Every command arising from the prediction imperative requires 
this pervasive real-world material “knowing and doing” presence. The new apparatus is the material 
expression of the prediction imperative, and it represents a new kind of power animated by the economic 
compulsion toward certainty. Two vectors converge in this fact: the early ideals of ubiquitous computing 
and the economic imperatives of surveillance capitalism. This convergence signals the metamorphosis of 
the digital infrastructure from a thing that we have to a thing that has us. 

Futuristic as this may sound, the vision of individuals and groups as so many objects to be 
continuously tracked, wholly known, and shunted this way or that for some purpose of which they are 
unaware has a history. It was coaxed to life nearly sixty years ago under the warm equatorial sun of the 
Galapagos Islands, when a giant tortoise stirred from her torpor to swallow a succulent chunk of cactus 
into which a dedicated scientist had wedged a small machine. 

It was a time when scientists reckoned with the obstinacy of free-roaming animals and concluded that 
surveillance was the necessary price of knowledge. Locking these creatures in a zoo would only eliminate 
the very behavior that scientists wanted to study, but how were they to be surveilled? The solutions once 
concocted by scholars of elk herds, sea turtles, and geese have been refurbished by surveillance 
capitalists and presented as an inevitable feature of twenty-first-century life on Earth. All that has changed 
is that now we are the animals. 


IT. The Tender Conquest of Unrestrained Animals 


It was a 1964 international expedition to the Galapagos Islands that presented a unique opportunity to 
explore telemetry, a frontier technology based on the long-distance transmission of computer data. A new 
breed of scientists who combined biology, physics, engineering, and electronics championed this new 
tech, and chief among these was R. Stuart MacKay, a physicist cum electrical engineer, biologist, and 
surgeon who was known among his scientific peers as the experts’ expert.2 

MacKay viewed telemetry as a means of enhancing and protecting the well-being of animal 
populations. A photo of MacKay from the Galapagos expedition shows him poised tenderly beside a giant 
tortoise that had swallowed his tiny machine; in another he gently holds a rare marine iguana with a 
sensor attached to its torso, all of it to measure the animals’ internal body temperatures. He emphasized 
the key element that distinguished telemetry from other forms of monitoring: the possibility of capturing 
behavior in its natural habitat with sensors of such compactness that they could disappear into the body 
without triggering the animal’s awareness: 


The use of a radio signal from a transmitter, in or on a subject, to carry information to a remote receiver for recording allows 
flexibility of movement and permits disturbance-free exploration of otherwise inaccessible parts of the body without the subject even 


being aware of the measuring process... the methods leave the subject in a relatively normal psychological and physiological state, 


and do not interfere with the continuation of normal activities.8 


MacKay’s published work focused primarily on the technical aspects of his studies, although 


occasionally there is a glimpse of larger purpose. Telemetry created the possibility of huge data sets and 
the opportunity for correlational studies at the scale of entire animal populations. He noted that the same 
techniques could be applied to the static world: forest canopies, the curing of concrete, chemical reaction 
vessels, and food processing. MacKay envisioned whole populations of connected data-emitting 
individuals. His first-generation “wearable technologies” made it possible to study “unrestrained 
animals” among every species, including people. Biomedical telemetry, he stressed, was uniquely suited 
to gather information that would be otherwise impossible to collect “in the wild.” The key principle was 
that his telematics operated outside an animal’s awareness. This was especially useful in solving 
problems such as the difficulty of measuring “uncooperative animals” and the need to gather data even 
when herds roamed through “inaccessible regions.” In other words, MacKay’s inventions enabled 
scientists to render animals as information even when they believed themselves to be free, wandering and 
resting, unaware of the incursion into their once-mysterious landscapes. 

MacKay stressed that the transmission and monitoring of sensor data were only part of the story. The 
route was not enough; it had to be the routing. He argued for a “reverse process” of telestimulation that 
would not only monitor behavior but also reveal how it could be modified and optimized, providing what 
he regarded as “a remote dialogue between the subject and the experimenter.”2 

MacKay’s broad vision has come to fruition in the digital age. Satellite acuity combined with the 
explosive growth of computational power fitted onto tiny slivers of silicon, advanced sensors, internet- 
enabled networks, and “big data” predictive analytics have produced extraordinary systems that reveal 
the meanings and movements of whole animal populations and their individuals: anywhere, anytime. The 
same wearables traveling on and in the bodies of animals have also become broad sensors of the planet’s 
climate, geography, and ecology, enabling “a quorum sensing of our planet, using a variety of species to 
tap into the diversity of senses that have evolved across animal groups,” producing a “sixth sense of the 
global animal collective.”!2 As you will already have guessed, there is little reason to suppose that these 
capabilities will remain trained on nonhuman species. 

Indeed, the threshold has already been crossed. In 2014 a team of University of Washington 
researchers led by Jenq-Neng Hwang announced a “super GPS” assembled from municipal surveillance 
cameras “to enable the dynamic visualization of the realistic situation of humans walking on the road and 
sidewalks, so eventually people can see the animated version of the real-time dynamics of city streets on 
a platform like Google Earth.”42 If this were a novel, then Professor MacKay’s brilliant work, along with 
that of the many dedicated scientists who followed in his footsteps, would serve as foreshadowing. 

In a metamorphosis that MacKay did not foresee, the science of animal tracking that grew from his 
pathbreaking vision became the template for surveillance capitalism’s next phase of evolution as 
telematics now applied to human behavior succumbed to the thrall of a new and lucrative logic of 
accumulation. The requirements of prediction that would later merge into an economic imperative were 
already evident in MacKay’s work. The need for economies of scope, both in extension and depth, is 
reflected in his foundational framework that aimed to render information about populations and the details 
of individuals, reaching into the distant corners of previously inaccessible regions. Today those regions 
include the front seat of your car, your kitchen, and your kidneys. His “reverse process” of telestimulation 
is resurrected in the economies of action that automatically stimulate behavior, not to save the human herd 
from catastrophe but rather to heighten the predictability of its behavior. 

MacKay yearned for discovery, but today’s “experimenters” yearn for certainty as they translate our 
lives into calculations. MacKay’s animals were unrestrained and innately uncooperative because they felt 
themselves to be free, sheltering and roaming in unknown terrain. Now, the un-self-conscious, easy 
freedom enjoyed by the human animal—the sense of being unrestrained that thrives in the mystery of 


distant places and intimate spaces—is simply friction on the path toward surveillance revenues. 


Il. Human Herds 


MacKay’s legacy is reimagined for our time in the work of Professor Joseph Paradiso of the MIT Media 
Lab, where some of surveillance capitalism’s most valuable capabilities and applications, from data 
mining to wearable technologies, were invented. 

Paradiso’s brilliant group of data scientists, engineers, musicians, and artists reconceives the world 
through the lens of Google Search by applying the same disciplines that mastered the web—datafication, 
indexing, browsing, and searching—to master reality itself. Without “ubiquitous sensate environments,” 
Paradiso writes, “the cognitive engines of this everywhere-enabled world are deaf, dumb, and blind, and 
can’t respond relevantly to the real-world events that they aim to augment.”/8 In other words, ubiquitous 
computing is meaningless without the ubiquitous sensing that conveys the experience for computation. 

To this end, Paradiso’s students invented a “ListenTree,” which emits streaming sound that “invites 
attention” and “points to a future where digital information might become a seamless part of the physical 
world.” He and his colleagues populated a 250-acre marsh with hundreds of sensors that measure and 
record temperature, humidity, moisture, light motion, wind, sound, tree sap flow, chemical levels, and 
more. They developed “inertial sensors” that track and compute complex movements and “flexible 
sensate fibers” to create “radically new functional substrates that can impact medicine, fashion, and 
apparel... and bring electronics into all things stretchable or malleable.” There are electronics that attach 
directly to skin in the form of tattoos and makeup, while fingernails and wrists are transformed into 
computational interfaces that can read finger gestures, even in the absence of hand movements. “Sensor 
tape” and “stickers” can adhere “to inaccessible surfaces and building materials,” where they can be 
“wirelessly interrogated....”14 

Paradiso and his colleagues wrestled with the paradox of, on the one hand, proliferating sensor data in 
nearly every environment—from smartphones to home devices to streets to cameras to cars—and, on the 
other hand, the difficulties involved in integrating sensor-generated data flows and producing meaningful 
analyses. Their answer was “DoppelLab,” a digital platform for combining and visually representing 
sensor data. The idea is to transform any physical space, from the interior of an office building to an 
entire city, into a “browse-able environment” where you can see and hear everything going on in that 
space as it flows from thousands or billions or trillions of sensors. Just as browsers like Netscape first 
“gave us access to the mass of data contained on the internet, so will software browsers enable us to 
make sense of the flood of sensor data that is on the way.”® 

The aim here is a grand synthesis: the collation and fusion of every sort of sensor data from every 
channel and device to develop a “virtual sensor environment” in which “crawlers will constantly traverse 
data... calculating state and estimating other parameters derived from the data” collected from 
everywhere from office interiors to entire cities. 

Paradiso is confident that “a proper interface to this artificial sensoria promises to produce... a 
digital omniscience... a pervasive everywhere augmented reality environment... that can be intuitively 
browsed” just as web browsers opened up the data contained on the internet. He insists that ubiquitous 
sensor information and computing will be “an extension of ourselves rather than an embodiment of an 
‘other.’” Information will stream “directly into our eyes and ears once we enter the age of wearables... 
the boundaries of the individual will be very blurry in this future.” 

According to Paradiso and his coauthor, Gershon Dublon, the next great technological challenge is 


“context aggregation,” which means the ability to assemble rapidly expanding sensor information into new 
“applications.” The idea is that every physical space and every trace of behavior within that space—bees 
buzzing, your smile, the temperature fluctuations in my closet, their breakfast conversation, the swoosh of 
the trees—will be “informated” (translated into information). Spaces can be aggregated into a seamless 
flow of searchable information, sights, and sounds in much the same way that Google once aggregated 
web pages for indexing and searching: “This shift will create a seamless nervous system that covers the 
planet—and one of the main challenges for the computing community now is how to merge the rapidly 
evolving ‘omniscient’ electronic sensoria onto human perception.” 2 

For all their brilliance, these creative scientists appear to be unaware of the restless economic order 
eager to commandeer their achievements under the flag of surveillance revenues. Paradiso does not 
reckon with the translation of his paradise of omniscience into the realpolitik of surveillance capitalism 
as the prediction imperative insists on surplus culled from these new flows and surveillance capitalists 
fill the front seats of the classroom of digital omniscience. 


IV. Surveillance Capitalism’s Realpolitik 


Waning levels of government leadership and funding for “ubiquitous computing” leave the technology 
companies to lead in basic research and applications, each vying to become “the Google” of the new 
apparatus and its architectures of extraction and execution.2 Despite the radical prospects of the 
ubiquitous connected sensate computational apparatus and the often-repeated claim “It will change 
everything,” technology firms in the US have, thus far, continued their run of relative lawlessness, 
unimpeded by any comprehensive social or regulatory vision. As Intel’s chief strategist for the “internet of 
things” commented in response to concerns over privacy implications, “One thing that we absolutely 
believe is that though we hear the conversation around policy, we don’t want policy to get in the way of 
technological innovation....”22 

In place of “policy” or a “social contract,” it is capitalism, and increasingly surveillance capitalism, 
that shapes the action. New behavioral futures markets and “targeted applications” are eagerly awaited. 
As Microsoft’s director of the machine intelligence platform for integrating and analyzing data from the 
“internet of things” says, “The part that’s equally cool and creepy is what happens after everybody and 
their competitor gets on board with smart devices: a big secondary market for data... a secondary revenue 
source.” These markets, he explains, are “just like Google or Facebook’s” markets for targeted 
advertising.”=! An IBM report concurs: “Thanks to the internet of things, physical assets are turning into 
participants in real-time global digital markets. The countless types of assets around us will become as 
easily indexed, searched and traded as any online commodity.... We call this the ‘liquification of the 
physical world.’”24 

In an ominous parallel to the rhetoric of “data exhaust” as the prelude to dispossession, this second 
phase of expropriation also requires new rhetoric that can simultaneously legitimate and distract from the 
real action unleashed by the prediction imperative. A new euphemism, “dark data,” plays this role. For 
example, Harriet Green directed IBM’s $3 billion investment in the “internet of things,” a resource 
commitment that aimed to make the company a serious contender to become “the Google” of ubiquitous 
computing. Green says that digital omniscience is impeded by the fact that most of the data collected by 
companies are “unstructured,” making them difficult to “datafy” and code.*? IBM’s customers are plagued 
by the question “What can we do with this [unstructured] data to make us more efficient or to create new 
products and services that we can sell to optimize what we’re doing or create new things for clients?”“4 


Unstructured data cannot merge and flow in the new circuits of liquefied assets bought and sold. They 
are friction. Green fixes on the declarative term that simultaneously names the problem and justifies its 
solution: dark data. The message we saw honed in the online world—“If you’re not in the system, you 
don’t exist”—is refined for this new phase of dispossession. Because the apparatus of connected things is 
intended to be everything, any behavior of human or thing absent from this push for universal inclusion is 
dark: menacing, untamed, rebellious, rogue, out of control. The stubborn expanse of dark data is framed 
as the enemy of IBM’s and its customers’ ambitions. Note the echoes of MacKay here, with his 
determination to penetrate the secrets of unrestrained animals and inaccessible regions. The tension is 
that no thing counts until it is rendered as behavior, translated into electronic data flows, and channeled 
into the light as observable data. Everything must be illuminated for counting and herding. 

In this way the notion of “dark data” handily becomes the “data exhaust” of ubiquitous computing. It 
provides the moral, technical, commercial, and legal rationale for powerful systems of machine 
intelligence that can capture and analyze behaviors and conditions never intended for a public life. For 
those who seek surveillance revenues, dark data represent lucrative and necessary territories in the 
dynamic universal jigsaw constituted by surveillance capitalism’s urge toward scale, scope, and action. 
Thus, the technology community casts dark data as the intolerable “unknown unknown” that threatens the 
financial promise of the “internet of things.” 

It is therefore understandable that Green portrays machine intelligence—and specifically IBM’s 
anthropomorphized artificial intelligence system called “Watson”—as the authoritative savior of an 
apparatus threatened by waste and incomprehensibility. Machine intelligence is referred to as “cognitive 
computing” at IBM, presumably to avoid the uneasy connotations of inscrutable power associated with 
words like machine and artificial. 

Under the leadership of CEO Ginni Rometty, the corporation invested heavily in “Watson,” heralded 
by the company as “the brains of the ‘internet of things.’” Rometty wants IBM to dominate the machine 
learning functions that will translate ubiquitous data into ubiquitous knowledge and action. “The first 
discussion is around how much dark data you have that only Watson and cognitive can really interrogate,” 
Green says. “You know the amount of data being created on a daily basis—much of which will go to 
waste unless it is utilized. This so-called dark data represents a phenomenal opportunity... the ability to 
use sensors for everything in the world to basically be a computer, whether it’s your contact lens, your 
hospital bed, or a railway track.”2° The message is that surveillance capitalism’s new instruments will 
render the entire world’s actions and conditions as behavioral flows. Each rendered bit is liberated from 
its life in the social, no longer inconveniently encumbered by moral reasoning, politics, social norms, 
rights, values, relationships, feelings, contexts, and situations. In the flatness of this flow, data are data, 
and behavior is behavior. The body is simply a set of coordinates in time and space where sensation and 
action are translated as data. All things animate and inanimate share the same existential status in this 
blended confection, each reborn as an objective and measurable, indexable, browsable, searchable “it.” 

From the vantage point of surveillance capitalism and its economic imperatives, world, self, and body 
are reduced to the permanent status of objects as they disappear into the bloodstream of a titanic new 
conception of markets. His washing machine, her car’s accelerator, and your intestinal flora are collapsed 
into a single dimension of equivalency as information assets that can be disaggregated, reconstituted, 
indexed, browsed, manipulated, analyzed, reaggregated, predicted, productized, bought, and sold: 
anywhere, anytime. 

The worldview elaborated by scientists such as Paradiso and business leaders such as Green has been 
swept into action on many fronts where digital omniscience is eagerly welcomed as the recipe for 
certainty in the service of certain profits. The next section is an opportunity to see this worldview in 


action, in a business sector far from the pioneers of surveillance capitalism: automobile insurance. 
Extraction and prediction become the hallmarks of a new logic of accumulation as insurers and their 
consultants plot their approach to surveillance revenues. In the plans and practices of these new actors, 
we witness both the determination to institutionalize economies of scope and action and the drift toward a 
dark new world in which the automatic and closely targeted means of behavioral modification are 
understood as the path to profit. 


V. Certainty for Profit 


In Chapter 3 we met Google’s Hal Varian, and now once again he lights the way, exposing the significance 
and specific requirements of the prediction imperative. Recall that Varian identified four new “uses” of 
the computer mediation of transactions.~“ The first of these was “data extraction and analysis,” from 
which we deduced the extraction imperative as one of the foundational mechanisms of surveillance 
capitalism. Varian says that the other three new uses—“new contractual forms due to better monitoring,” 
“personalization and customization,” and “continuous experiments”—“will, in time, become even more 
important than the first.”28 That time has come. 

“Because transactions are now computer-mediated we can observe behavior that was previously 
unobservable and write contracts on it,” Varian says. “This enables transactions that were simply not 
feasible before.” He gravitates to the example of “vehicular monitoring systems,” recognizing their 
paradigmatic power. Varian says that if someone stops making monthly car payments, “Nowadays it’s a 
lot easier just to instruct the vehicular monitoring system not to allow the car to be started and to signal 
the location where it can be picked up.”“2 Insurance companies, he notes, can also rely on these 
monitoring systems to check if customers are driving safely and thus determine whether to maintain the 
insurance policy, vary the cost of premiums, and decide whether to pay a claim. 

Varian’s new uses of computer mediation in this insurance realm are entirely dependent upon internet- 
enabled devices that know and do. In fact, they are impossible to imagine without the material means of 
extraction and execution architectures planted in and permeating the real world. The vehicular monitoring 
system that he prescribes, for example, provides economies of scope and action. It knows and intervenes 
in the state of play, monitoring data and acting on programmed instructions to shut off the car’s engine, 
thus allowing the repo man to locate the disabled automobile and its vanquished driver. 

As the prediction imperative pulls supply operations into the real world, product or service providers 
in established sectors far from Silicon Valley are enthralled by the prospects of surveillance revenues. 
For example, the CEO of Allstate Insurance wants to be like Google: “There are lots of people who are 
monetizing data today. You get on Google, and it seems like it’s free. It’s not free. You’re giving them 
information; they sell your information. Could we, should we, sell this information we get from people 
driving around to various people and capture some additional profit source...? It’s a long-term game.”22 
Automobile insurers appear to be especially eager to implement Varian’s vision and MacKay’s 
telematics. The fight for your car, it turns out, is an illustration of the intensity of purpose with which 
companies great and small now pursue behavioral surplus. 

Auto insurers have long known that risk is highly correlated with driver behavior and personality, but 
there was little that they could do about it.2! Now the remote sensate monitoring systems of modern 
telematics can provide a continuous stream of data about where we are, where we’re going, the details of 
our driving behavior, and the conditions of our vehicle. App-based telematics can also calculate how we 
are feeling and what we are saying by integrating dashboard and even smartphone information. 


Auto insurers are besieged by consultants and would-be technology partners who proffer surveillance 
capitalistic strategies that promise a new chapter of commercial success. “Uncertainty will be strongly 
reduced,” intones a McKinsey report on the future of the insurance industry. “This leads to 
demutualization and a focus on predicting and managing individual risks rather than communities.”22 A 
report by Deloitte’s Center for Financial Services counsels “risk minimization’—a euphemism for 
guaranteed outcomes—through monitoring and enforcing policyholder behavior in real time, an approach 
called “behavioral underwriting.” “Insurers can monitor policyholder behavior directly,” Deloitte 
advises, by “recording the times, locations, and road conditions when they drive, whether they rapidly 
accelerate or drive at high or even excessive speeds, how hard they brake, as well as how rapidly they 
make turns and whether they use their turn signals.”22 Telematics produce continuous data flows, so real- 
time behavioral surplus can replace the traditional “proxy factors,” such as demographic information, that 
had previously been used to calculate risk. This means that surplus must be both plentiful (economies of 
scale) and varied (economies of scope) in both range and depth. 

Even smaller underwriters that cannot afford extensive capital outlays for telematics are advised that 
they can accomplish most of these aims with a smartphone application, eliminating costly hardware and 
data-transmission expenses: “These insurers may also benefit because a mobile app gathers first-hand 
data on the behavior and performance of the driver carrying the smartphone... yielding a 360-degree view 
of the total exposure being underwritten....”=4 

As certainty replaces uncertainty, premiums that once reflected the necessary unknowns of everyday 
life can now rise and fall from millisecond to millisecond, informed by the precise knowledge of how fast 
you drive to work after an unexpectedly hectic early morning caring for a sick child or if you perform 
wheelies in the parking lot behind the supermarket. “We know that 16-year old drivers have a whole lot 
of accidents... but not every 16-year old is a lousy driver,” observes one insurance industry telematics 
expert. Rates based on actual behavior are “a big advantage in being able to price appropriately.”=? This 
kind of certainty means that insurance contracts designed to mitigate risk now give way to machine 
processes that respond “almost immediately” to nuanced infractions of prescribed behavioral parameters 
and thus substantially decrease risk or eliminate it entirely.*° 

Telematics are not intended merely to know but also to do (economies of action). They are hammers; 
they are muscular; they enforce. Behavioral underwriting promises to reduce risk through machine 
processes designed to modify behavior in the direction of maximum profitability. Behavioral surplus is 
used to trigger punishments, such as real-time rate hikes, financial penalties, curfews, and engine 
lockdowns, or rewards, such as rate discounts, coupons, and gold stars to redeem for future benefits. The 
consultancy firm AT Kearney anticipates “IoT enriched relationships” to connect “more holistically” with 
customers “to influence their behaviors.”=2 

Varian’s blithe statement that “it’s a lot easier” to instruct a vehicular monitoring system to shut off a 
car when a payment is late is not hyperbole. For example, Spireon, which describes itself as “the largest 
aftermarket vehicle telematics company” and specializes in tracking and monitoring vehicles and drivers 
for a variety of customers such as lenders, insurers, and fleet owners, offers a system akin to Varian’s 
ideal.*8 Its “LoanPlus Collateral Management System” pushes alerts to drivers when they have fallen 
behind in their payments, remotely disables the vehicle when delinquency exceeds a predetermined 
period, and locates the vehicle for the repo man to recover. 

Telematics announce a new day of behavioral control. Now the insurance company can set specific 
parameters for driving behavior. These can include anything from fastening the seat belt to rate of speed, 
idling times, braking and cornering, aggressive acceleration, harsh braking, excessive hours on the road, 
driving out of state, and entering a restricted area.22 These parameters are translated into algorithms that 


continuously monitor, evaluate, and rank the driver, calculations that translate into real-time rate 
adjustments. 

According to a patent held by Spireon’s top strategist, insurers can eliminate uncertainty by shaping 
behavior.“ The idea is to continuously optimize the insurance rate based on monitoring the driver’s 
adherence to behavioral parameters defined by the insurer. The system translates its behavioral 
knowledge into power, assigning credits or imposing punishments on drivers. Surplus is also translated 
into prediction products for sale to advertisers. The system calculates “behavioral traits” for advertisers 
to target, sending ads directly to the driver’s phone. A second patent is even more explicit about triggers 
for punitive measures.*! It identifies a range of algorithms that activate consequences when the system’s 
parameters are breached: “a violation algorithm,” “a curfew algorithm,” “a monitoring algorithm,” “an 
adherence algorithm,” “a credit algorithm.” 

The consultancy firms are aligned in advising all their insurance clients to get into the surveillance 
game. AT Kearney acknowledges that the “connected car” is a proving ground for what is to come: 
“Ultimately, IoT’s true value depends on customers adjusting their behaviors and risk profiles based on 
feedback from their ‘things.’”42 Health insurers are another target: “Wearable accelerometers” could 
“improve traceability of their compliance” with prescribed exercise regimes, and “digestible sensors” 
could track compliance with dietary and medication schedules, “providing higher truth and better 
granularity than a monthly refill.” 

Deloitte acknowledges that according to its own survey data, most consumers reject telematics on the 
basis of privacy concerns and mistrust companies that want to monitor their behavior. This reluctance can 
be overcome, the consultants advise, by offering cost savings “significant enough” that people are willing 
“to make the [privacy] trade-off,” in spite of “lingering concerns....” If price inducements don’t work, 
insurers are counseled to present behavioral monitoring as “fun,” “interactive,” “competitive,” and 
“oratifying,” rewarding drivers for improvements on their past record and “relative to the broader policy 
holder pool.”“ In this approach, known as “gamification,” drivers can be engaged to participate in 
“performance based contests” and “incentive based challenges.” 

If all else fails, insurers are advised to induce a sense of inevitability and helplessness in their 
customers. Deloitte counsels companies to emphasize “the multitude of other technologies already in play 
to monitor driving” and that “enhanced surveillance and/or geo-location capabilities are part of the world 
we live in now, for better or worse.”“2 

Behavioral underwriting offers auto insurers cost savings and efficiencies, but it is not the endgame 
for a revitalized insurance industry. The analytics that produce targeted advertising in the online world 
are repurposed for the real world, laying the foundation for new behavioral futures markets that trade in 
predictions of customer behavior. This is where the real drive for surveillance revenues is focused. For 
example, an executive of cloud services provider Covisint advises clients aiming to “cash in” on 
automotive telematics to move beyond targeted ads to “targeted applications.” These are not ads on a 
screen but real-life experiences shaped by the same capabilities as targeted ads and designed to lure you 
into real places for the sake of others’ profit. That means selling driver data to third parties that will 
figure out where you are, where you’re going, and what you want: “They know what restaurants you like 
because you drive your car there, so they can recommend restaurants as you’re driving and the restaurants 
will pay....”4 

Behavioral surplus is understood as the raw material for products that establish “co-marketing” with 
other services such as “towing, auto repair, car washes, restaurants, retail outlets... .”48 The consultants at 
McKinsey make a similar recommendation, advising insurers that the “internet of things” enables their 
expansion into “completely new areas” such as “data market-places.” Health surplus can be “monetized,” 
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says Deloitte, by providing “relevant referrals.” The firm advises its clients, especially those unlikely to 
reach scale in telematics, to establish partnerships with “digital players.”“2 The model is a 2016 deal 
between IBM and General Motors that announced the creation of “OnStar Go,” the car industry’s “first 
cognitive mobility platform.” Dell and Microsoft have launched “internet of things” insurance 
“accelerators.” Dell provides insurers with hardware, software, analytics, and services to “more 
accurately predict risk and take preventative measures,” and Microsoft has linked up with American 
Family Insurance to develop startups focused on home automation.22 

The data companies were once regarded as mere “suppliers,” but it is more likely that the auto 
companies will become suppliers to the data behemoths. “Google tries to accompany people throughout 
their day, to generate data and then use that data for economic gain,” acknowledges Daimler’s CEO. “It’s 
at that point where a conflict with Google seems preprogrammed.”2! Google and Amazon are already 
locked in competition for the dashboard of your car, where their systems will control all communication 
and applications. From there it is a short step to telemetry and related data. Google already offers 
applications developers a cloud-based “scaleable geolocation telemetry system’ using Google Maps. In 
2016 Google France announced its interest in partnerships with insurance companies “to develop bundles 
of products which blend technology and hardware with insurance.” That same year a report from Cap 
Gemini consultants found that 40 percent of insurers see Google “as a potential rival and threat because of 
its strong brand and ability to manage customer data.” 


VI. Executing the Uncontract 


These examples drawn from the ordinary world of automobile insurance teach some extraordinary 
lessons. Drivers are persuaded, induced, incentivized, or coerced into a quid pro quo that links pricing to 
the expansion of a real-world extraction/execution architecture aimed at new behavioral surplus flows 
(economies of scope). Behavioral data drawn from their experience are processed, and the results flow in 
two directions. First, they return to the drivers, executing procedures to interrupt and shape behavior in 
order to enhance the certainty, and therefore profitability, of predictions (economies of action). Second, 
prediction products that rank and sort driver behavior flow into newly convened behavioral futures 
markets in which third parties lay bets on what drivers will do now, soon, and later: Will he maintain a 
high safety rating? Will she act in compliance with our rules? Will he drive like a girl? These bets 
translate into pricing, incentive structures, and monitoring and compliance regimes. In both operations, 
surplus drawn from the driver’s experience is repurposed as the means to shape and compel the 
driver’s experience for the sake of guaranteed outcomes. Most of this occurs, as MacKay advised, 
outside the driver’s awareness while she still thinks that she is free. 

The Google declarations underwrite all the action here. As Varian writes, “Because transactions are 
now computer-mediated we can observe behavior that was previously unobservable and write contracts 
on it. This enables transactions that were simply not feasible before.”=° Varian’s “we” refers to those 
with privileged access to the shadow text into which behavioral data flow. Our behavior, once 
unobservable, is declared as free for the taking, theirs to own, and theirs to decide how to use and how to 
profit from. This includes the production of “new contractual forms” that compel us in ways that would 
not have been possible but for surveillance capitalism’s original declarations of dispossession. 

Varian recognized that the subregions of automotive telematics exemplify this new economic frontier 
when he wrote, “Nowadays it’s a lot easier just to instruct the vehicular monitoring system not to allow 
the car to be started and to signal the location where it can be picked up.”=4 Yawn. But wait. “A lot 


easier” for whom? He means, of course, a lot easier for the “we” that now observes what was, until 
surveillance capitalism, unobservable and executes actions that were, until surveillance capitalism, not 
feasible. Varian’s laid-back, simple prose is a kind a lullaby that makes his observations seem banal, so 
ordinary as to barely warrant comment. But in Varian’s scenario, what happens to the driver? What if 
there is a child in the car? Or a blizzard? Or a train to catch? Or a day-care center drop-off on the way to 
work? A mother on life support in the hospital still miles away? A son waiting to be picked up at school? 

It was not long ago that Varian’s prosaic proclamations were regarded as the stuff of nightmares. In his 
1967 book The Year 2000 the hyper-rational wunderkind futurist Herman Kahn anticipated many of the 
capabilities that Varian now assigns to the new extraction/execution architecture.22 Kahn was no shrinking 
violet. He was rumored to be director Stanley Kubrick’s model for the title character in Dr. Strangelove, 
and he was well-known for arguing that nuclear war is both “winnable” and “survivable.” Yet it was 
Kahn who foresaw innovations such as Varian’s vehicular monitoring system and characterized them as “a 
twenty-first century nightmare.” Among his many technology-oriented insights, Kahn foresaw automated 
computer systems that track all vehicular movements and also listen to and record conversations with all 
the capability available for high-speed scan and search. He imagined computer systems able to detect and 
respond to individual behavior—a raised voice, a threatening tone: “Such computers may also be able to 
apply a great deal of inferential logic on their own—they may become a sort of transistorized Sherlock 
Holmes making hypotheses and investigating leads in a more or less autonomous or self-motivated 
manner....”2° Anyone who wields this kind of knowledge, he concluded, is, like Faust, “less immoral than 
amoral... indifferent to the fate of those who stand in his way rather than brutal.”22 

Contemporary reviewers of Kahn’s book invariably seized upon the dark “nightmare scenarios” of the 
computerized surveillance theme, the science-fiction—like forms of control that, they assumed, “will be 
actively feared and resented by many.”22 Despite the wide range of scenarios that Kahn presented in his 
book on the distant year 2000, Kahn’s voyage into the “unthinkable” was viewed by the public as a way to 
prepare for “the worst possible outcome” in a terrifying “nightmare of social controls.”°2 Yet now that 
same nightmare is rendered as an enthusiastic progress report on surveillance capitalism’s latest triumphs. 
Varian’s update is delivered without self-consciousness or a hint of controversy, rather than the 
astonishment and revulsion that were predicted just decades ago. How has the nightmare become banal? 
Where is our sense of astonishment and outrage? 

Political scientist Langdon Winner grappled with this question in his seminal book Autonomous 
Technology, published in 1977. His answer? “What we lack is our bearings,” he wrote. Winner 
painstakingly described the ways in which our experience of “things technological” confounds “our 
vision, our expectations, and our capacity to make intelligent judgments. Categories, arguments, 
conclusions, and choices that would have been entirely obvious in earlier times are obvious no longer.” 

So let us establish our bearings. What Varian celebrates here is not a new form of contract but rather 
a final solution to the enduring uncertainty that is the raison d’étre of “contract” as a means of “private 
ordering.” In fact, the use of the word contract in Varian’s formulation is a perfect example of the 
horseless-carriage syndrome. Varian’s invention is unprecedented and cannot be understood as simply 
another kind of contract. It is, in fact, the annihilation of contract; this invention is better understood as 
the uncontract. 

The uncontract is a feature of the larger complex that is the means of behavioral modification, and it is 
therefore an essential modality of surveillance capitalism. It contributes to economies of action by 
leveraging proprietary behavioral surplus to preempt and foreclose action alternatives, thus replacing the 
indeterminacy of social processes with the determinism of programmed machine processes. This is not 
the automation of society, as some might think, but rather the replacement of society with machine action 


dictated by economic imperatives. 

The uncontract is not a space of contractual relations but rather a unilateral execution that makes those 
relations unnecessary. The uncontract desocializes the contract, manufacturing certainty through the 
substitution of automated procedures for promises, dialogue, shared meaning, problem solving, dispute 
resolution, and trust: the expressions of solidarity and human agency that have been gradually 
institutionalized in the notion of “contract” over the course of millennia. The uncontract bypasses all that 
social work in favor of compulsion, and it does so for the sake of more-lucrative prediction products that 
approximate observation and therefore guarantee outcomes. 

This substitution of machine work for social work is possible thanks to the success of Google’s 
declarations and the road that Google paved for surveillance capitalists’ dominance of the division of 
learning. Sitting in the catbird seat, Google can observe what was previously unobservable and know 
what was previously unknowable. As a result, the company can do what was previously undoable: bypass 
social relations in favor of automated machine processes that compel the behaviors that advance 
commercial objectives. When we celebrate the uncontract, as Varian and others do, we celebrate the 
asymmetries of knowledge and power that produce these new possibilities. The uncontract is a signpost 
that reminds us of our bearings as we follow the next sections of this chapter toward a clearer picture of 
surveillance capitalism’s growing ambitions in the annexation of “reality” to its kingdom of conquered 
human experience. 


VII. Inevitabilism 


It is difficult to keep your bearings when everyone around you is losing theirs. The transition to ubiquitous 
computing, “when sensors are everywhere,” Paradiso writes, won’t be “incremental” but rather “a 
revolutionary phase shift much like the arrival of the world wide web.”©! The same “phase shift” that is 
understood by its architects as the universal antidote to uncertainty is anticipated with absolute certainty. 
Paradiso is not alone here. On the contrary, the rhetoric of inevitability is so “ubiquitous” that within the 
tech community it can be considered a full-blown ideology of inevitabilism. 

The sense of incontestable certainty that infuses Paradiso’s vision has long been recognized as a key 
feature of utopianism. In their definitive history of utopian thought, Frank and Fritzie Manuel wrote that 
“since the end of the eighteenth century the predictive utopia has become a major form of imaginative 
thought and has preempted certain scientific techniques of forecasting... the contemporary utopia... binds 
past, present, and future together as though fated. The state they depict appears virtually ordained either 
by god or by history; there is a carry-over of millenarian certainty...” 

The Manuels, along with many other historians, consider Marxism to be the last great modern utopia.© 
There are hundreds of passages in Karl Marx’s writing that convey his inevitabilism. In the very first 
section of The Communist Manifesto, published in 1848, Marx wrote the following: “What the 
bourgeoisie, therefore, produces, above all, is its own grave-diggers. Its fall and the victory of the 
proletariat are equally inevitable.”™ 

Before the rise of the modern utopia, the genre was largely composed of fantastical narratives in 
which isolated pockets of human perfection were discovered in exotic mountain aeries, hidden valleys, or 
faraway islands. Modern utopias such as Marxism diverge from those fairy tales, addressing “the 
reformation of the entire species” with a rational systemic vision “whose province was the whole 
world.” No longer content as mere weavers of dreams, modern utopianists shifted toward totalistic and 
universal visions, prophecies of “the ineluctable end toward which mankind was moving.”® 


Now the proselytizers of ubiquitous computing join Marx and other modern utopianists in postulating a 
new phase of history, like Paradiso’s “revolutionary phase shift,” in which all of society is reassembled 
in a novel and superior pattern. Despite the fact that inevitability is the opposite of politics and history, 
apostles of the apparatus routinely hijack historical metaphors that lend a veneer of gravitas to their 
claims. The rise of the apparatus is alternatively cast as the inauguration of a new “age,” “era,” “wave,” 
“phase,” or “stage.” This kind of historical framing conveys the futility of opposition to the categorical 
inevitability of the march toward ubiquity. 

Silicon Valley is the axis mundi of inevitabilism. Among high-tech leaders, within the specialist 
literature, and among expert professionals there appears to be universal agreement on the idea that 
everything will be connected, knowable, and actionable in the near future: ubiquity and its consequences 
in total information are an article of faith. 

Not surprisingly, Google’s leaders are passionate inevitabilists. The very first sentences of Schmidt 
and Cohen’s 2013 book, The New Digital Age, exemplify this thrust: “Soon everyone on Earth will be 
connected,” they write. So-called predictive “laws” such as “Moore’s Law” and “photonics” are called 
upon to signal this new iron law of necessity that will produce exponential growth in connectivity and 
computational power.® And later, “The collective benefit of sharing human knowledge and creativity 
grows at an exponential rate. In the future, information technology will be everywhere, like electricity. It 
will be a given.”©2 When the book’s assertions garnered some criticism, the authors confronted their 
critics in an afterword to the paperback edition: “But bemoaning the inevitable increase in the size and 
reach of the technology sector distracts us from the real question.... Many of the changes we discuss are 
inevitable. They’re coming.” 

Despite its pervasiveness both in Silicon Valley and in the wider culture of data scientists and 
technology developers, inevitabilism is rarely discussed or critically evaluated. Paradiso’s conception of 
a “digital omniscience” is taken for granted, with little discussion of politics, power, markets, or 
governments. As in most accounts of the apparatus, questions of individual autonomy, moral reasoning, 
social norms and values, privacy, decision rights, politics, and law take the form of afterthoughts and 
genuflections that can be solved with the correct protocols or addressed with still more technology 
solutions. If information will stream “directly into our eyes and ears” and “the boundaries of the 
individual will be very blurry,” then who can access that information? What if I don’t want my life 
streaming through your senses? Who knows? Who decides? Who decides who decides? The answers to 
such questions are drowned in the thrum of all things continuously illuminated, registered, counted, 
controlled, and judged. 

The best that Paradiso can offer up is a suggestion that “the law could give a person ownership or 
control of data generated in his or her vicinity; a person could then choose to encrypt or restrict those data 
from entering the network.”® Paradiso imagines a society in which it falls to each individual to protect 
herself from the omniscient ubiquitous sensate computational systems of the new apparatus. Rather than 
paradise, it seems a recipe for a new breed of madness. Yet this is precisely the world that is now under 
construction around us, and this madness appears to be a happy feature of the plan. 

Between 2012 and 2015, I interviewed 52 data scientists and specialists in the “internet of things.” 
They came from 19 different companies with a combined 586 years of experience in high-technology 
corporations and startups, primarily in Silicon Valley. I spoke with them about the prominence of 
inevitability rhetoric among the purveyors of the new apparatus, and I posed the same question to each 
one: why do so many people say that ubiquitous computing is inevitable? The agreement among their 
responses was striking. Although they did not have the language of surveillance capitalism, nearly every 
interviewee regarded inevitability rhetoric as a Trojan horse for powerful economic imperatives, and 


each one of them lamented the lack of any critical discussion of these assumptions. 

As the marketing director of a Silicon Valley firm that sells software to link smart devices told me, 
“There’s all that dumb real estate out there and we’ve got to turn it into revenue. The ‘internet of things’ is 
all push, not pull. Most consumers do not feel a need for these devices. You can say ‘exponential’ and 
‘inevitable’ as much as you want. The bottom line is that the Valley has decided that this has to be the next 
big thing so that firms here can grow.” 

I spoke with a senior engineer from a large tech company that invests heavily in the “internet of 
things.” The response: 


Imagine you have a hammer. That’s machine learning. It helped you climb a grueling mountain to reach the summit. That’s machine 
learning’s dominance of online data. On the mountaintop you find a vast pile of nails, cheaper than anything previously imaginable. 
That’s the new smart sensor tech. An unbroken vista of virgin board stretches before you as far as you can see. That’s the whole 
dumb world. Then you learn that any time you plant a nail in a board with your machine learning hammer, you can extract value from 
that formerly dumb plank. That’s data monetization. What do you do? You start hammering like crazy and you never stop, unless 


somebody makes you stop. But there is nobody up here to make us stop. This is why the “internet of everything” is inevitable. 


A senior systems architect laid out the imperative in the clearest terms: “The IoT is inevitable like 
getting to the Pacific Ocean was inevitable. It’s manifest destiny. Ninety-eight percent of the things in the 
world are not connected. So we’re gonna connect them. It could be a moisture temperature that sits in the 
ground. It could be your liver. That’s your IoT. The next step is what we do with the data. We’11 visualize 
it, make sense of it, and monetize it. That’s our IoT.” 


VII. Men Made It 


The relentless drumbeat of inevitabilist messages presents the new apparatus of ubiquity as the product of 
technological forces that operate beyond human agency and the choices of communities, an implacable 
movement that originates outside history and exerts a momentum that in some vague way drives toward 
the perfection of the species and the planet. The image of technology as an autonomous force with 
unavoidable actions and consequences has been employed across the centuries to erase the fingerprints of 
power and absolve it of responsibility. The monster did it, not Victor Frankenstein. However, the ankle 
bracelet does not monitor the prisoner; the criminal justice system does that. 

Every doctrine of inevitability carries a weaponized virus of moral nihilism programmed to target 
human agency and delete resistance and creativity from the text of human possibility. Inevitability rhetoric 
is a cunning fraud designed to render us helpless and passive in the face of implacable forces that are and 
must always be indifferent to the merely human. This is the world of the robotized interface, where 
technologies work their will, resolutely protecting power from challenge. 

No one has expressed this with more insight and economy than John Steinbeck in the opening chapters 
of his masterwork, The Grapes of Wrath, which describes the dustbow! farmers who are thrown out of 
their Oklahoma homes during the Great Depression and then head west to California. The families are 
forced off the land that they have tended for generations. They plaintively argue their case to the bank 
agents sent to impress upon them the facts of their helplessness. But the agents respond with “The bank is 
something else than men. It happens that every man in a bank hates what the bank does, and yet the bank 
does it. The bank is something more than men, I tell you. It’s the monster. Men made it, but they can’t 
control it.”® 


This theme of supposed technological autonomy is a venerable one among technology scholars. 
Langdon Winner again proves to be a worthy guide when he reminds us that an unquestioning acceptance 
of technology has become a feature of modern life: “The changes and disruptions that an evolving 
technology repeatedly caused in modern life were accepted as given or inevitable simply because no one 
bothered to ask whether there were other possibilities.” 

Winner observes that we have allowed ourselves to become “committed” to a pattern of technological 
“drift,” defined as “accumulated unanticipated consequences.” We accept the idea that technology must 
not be impeded if society is to prosper, and in this way we surrender to technological determinism. 
Rational consideration of social values is considered “retrograde,” Winner writes, “not the ticket that 
scientific technology gives to civilization.... To this day, any suggestions that the forward flow of 
technological innovation be in any way limited... violate a fundamental taboo.... Instead we accept the 
change, later looking back upon what we have done to ourselves as a topic of curiosity.”2 To Winner’s 
“curiosity” I add another theme: remorse. 

Surveillance capitalist leaders assume that we will succumb to the naturalistic fallacy as Steinbeck’s 
farmers were meant to do. Because Google is successful—because surveillance capitalism is successful 
—its rules must obviously be right and good. Like the bank agents, Google wants us to accept that its rules 
simply reflect the requirements of autonomous processes, something that people cannot control. However, 
our grasp of the inner logic of surveillance capitalism suggests otherwise. Men and women made it, and 
they can control it. They merely choose not to do so. 

Inevitabilism enshrines the apparatus of ubiquity as progress but conceals the realpolitik of 
surveillance capitalism at work behind the scenes. We know that there can be alternative paths to a robust 
information capitalism that produces genuine solutions for a third modernity. We have seen that 
surveillance capitalism was discovered and honed in history, handcrafted by men and women to serve the 
interests of impatient capital. It is this same logic that now demands ubiquity, ready to colonize technical 
developments for the sake of its imperatives and growth. Inevitabilism operates in the service of these 
imperatives as it distracts attention from the ambitions of a rising economic order and the competitive 
anxieties that drive the surveillance project toward certainty, thus necessitating its ever-more-voracious 
claims on our behavior. 

Inevitabilism precludes choice and voluntary participation. It leaves no room for human will as the 
author of the future. This raises questions: At what point does inevitabilism’s claim to ubiquitous 
extraction and execution shade into abuse? Will inevitabilism’s utopian declarations summon new forms 
of coercion designed to quiet restless populations unable to quell their hankering for a future of their 
choice? 


IX. To the Ground Campaign 


Google’s declarations; surveillance capitalism’s dominance over the division of learning in society and 
its laws of motion; ubiquitous architectures of extraction and execution; MacKay’s penetration of 
inaccessible regions while observing unrestrained animals with methods that elude their awareness; the 
uncontract and its displacement of society; Paradiso’s ubiquitous sensate environment; dark data; the 
inevitabilism evangelists: there is one place where all these elements come together and transform a 
shared public space built for human engagement into a petri dish for the reality business of surveillance 
capitalism. That place is the city. 

Cisco has 120 “smart cities” globally, some of which have embraced Cisco Kinetic, which as Jahangir 


Mohammed, the company’s vice president and general manager of IoT, explains in a blog post, “is a 
cloud-based platform that helps customers extract, compute, and move data from connected things to IoT 
applications to deliver better outcomes.... Cisco Kinetic gets the right data to the right applications at the 
right time... while executing policies to enforce data ownership, privacy, security and even data 
sovereignty laws.” But, as is so often the case, the most audacious effort to transform the urban 
commons into the surveillance capitalist’s equivalent of Paradiso’s 250-acre marsh comes from Google, 
which has introduced and legitimated the concept of the “for-profit city.” Just as MacKay had counseled 
and Weiser proselytized, the computer would be operational everywhere and detectable nowhere, always 
beyond the edge of individual awareness. 

In 2015, shortly after Google reorganized itself into a holding company called Alphabet, Sidewalk 
Labs became one of nine “confirmed companies” under the Alphabet corporate umbrella. Whether what 
even Sidewalk CEO Dan Doctoroff, a former private equity financier, CEO of Bloomberg, and deputy 
mayor of New York City in the Bloomberg administration, refers to as a “Google city” succeeds, the 
company has interested the public by recasting our central gathering place as a commercial operation in 
which once-public assets and functions are reborn as the cornered raw materials earmarked for a new 
marketplace. In this vision, MacKay and Paradiso’s conceptions come to fruition under the auspices of 
surveillance capitalism in a grand scheme of vertically integrated supply, production, and sales. 

Sidewalk Labs’ first public undertaking was the installation of several hundred free internet-enabled 
kiosks in New York City, ostensibly to combat the problem of “digital inequality.” As we saw with 
Google Street View, the company can siphon a lot of valuable information about people from a Wi-Fi 
network, even if they don’t use the kiosks.4 Doctoroff has characterized the Sidewalk Labs’ kiosks as 
“fountains of data” that will be equipped with environmental sensors and also collect “other data, all of 
which can create very hyperlocal information about conditions in the city.” 

In 2016 the US Department of Transportation (DOT) announced a partnership with Sidewalk Labs “to 
funnel transit data to city officials.” The DOT worked to draw cities into Google’s orbit with a 
competition for $40 million in grants. Winners would work with Sidewalk Labs to integrate technology 
into municipal operations, but Sidewalk Labs was eager to work with finalists in order to develop its 
own traffic-management system, Flow.2 Flow relies on Google Maps, Street View vehicles, and machine 
intelligence to capture and analyze data from drivers and public spaces.“ These analyses produce 
prediction products described as “inferences about where people are coming from or going,” enabling 
administrators “to run virtual experiments” and improve traffic flow. 

Doctoroff postulates a city presided over by digital omniscience: “We’re taking everything from 
anonymized smartphone data from billions of miles, trips, sensor data, and bringing that into a 
platform.” Sidewalk refers to its high-tech services as “new superpowers to extend access and 
mobility.” Algorithms designed to maintain critical behaviors within a prescribed zone of action would 
manage these data flows: “In a world in which we can monitor things like noise or vibrations, why do we 
need to have these very prescriptive building codes?” As an alternative, Doctoroff suggests 
“performance-based zoning” administered by the ubiquitous apparatus through the medium of algorithms. 
These processes, like Varian’s vehicular monitoring systems, are indifferent to why you behave as long as 
they can monitor and control the behavior you produce. As Doctoroff explains it, “I don’t care what you 
put here as long as you don’t exceed performance standards like noise levels....” This is preferable, he 
says, because it enhances “the free flow of property... that is a logical extension of... these 
technologies.”“2 Why should citizens have any say over their communities and the long-term implications 
of how luxury high-rises, hotels, or a residential building going commercial could affect rents and local 
businesses as long as an algorithm is satisfied with noise thresholds? 


When Columbus, Ohio, was named winner of the DOT competition, it began a three-year 
demonstration project with Sidewalk, including a hundred kiosks and free access to the Flow software. 
Documents and correspondence from this collaboration eventually obtained by the Guardian describe 
innovations such as “dynamic parking,” “optimized parking enforcement,” and a “shared mobility 
marketplace” that reveal a more troubling pattern than the rhetoric suggests. Sidewalk’s data flows 
combine public and private assets for sale in dynamic, real-time virtual markets that extract maximum 
fees from citizens and leave municipal governments dependent upon Sidewalk’s proprietary information. 
For example, public and private parking spaces are combined in online markets and rented “on demand” 
as the cost of parking varies in real time, substantially increasing parking income. Optimized parking 
enforcement depends on Sidewalk’s algorithms “to calculate the most lucrative routes for parking cops,” 
earning cities millions of extra dollars that they desperately need but that arrive at the expense of their 
citizens. 

Cities are required to invest substantial public monies in Sidewalk’s technology platform, including 
channeling municipal funds earmarked for low-cost public bus service into “mobility markets” that rely 
on private ride-sharing companies such as Uber. The company insists that cities “share public transport 
data with ride-sharing companies, allowing Uber to direct cars to overcrowded bus stops.” The Flow 
Transit system integrates information and payment for nearly every kind of transport into Google Maps, 
and cities are obligated to “upgrade” to Sidewalk’s mobile payment system “for all existing transit and 
parking services.” Just as it requires public-transit data, Sidewalk also insists that cities share all parking 
and ridership information with Sidewalk Labs in real time.2? When asked, Doctoroff has emphasized the 
novel blending of public functions and private gain, assuring his listeners on both counts that “our mission 
is to use technology to change cities... to bring technology to solve big urban problems.... We expect to 
make a lot of money from this.”= 

In April 2016 a “curated group of leaders” in tech, media, and finance met at the Yale Club in 
Manhattan to hear Sidewalk CEO Dan Doctoroff’s talk: “Google City: How the Tech Juggernaut Is 
Reimagining Cities—Faster Than You Realize.”®2 His remarks provide a candid assessment of the 
“Google city” as a market operation shaped by the prediction imperative. He could not have been more 
direct in articulating Sidewalk Labs’ approach as a translation of Google’s online world to the reality of 
city life: 


In effect, what we’re doing is replicating the digital experience in physical space.... So ubiquitous connectivity; incredible computing 
power including artificial intelligence and machine learning; the ability to display data; sensing, including cameras and location data as 
well as other kinds of specialized sensors.... We fund it all... through a very novel advertising model.... We can actually then target 
ads to people in proximity, and then obviously over time track them through things like beacons and location services as well as their 

browsing activity. 22 

Later that year, Sidewalk announced collaborations with sixteen additional cities, noting that achieving 
scale would enable it to improve its Flow software products. Doctoroff referred to these collaborations 
as “inevitable.” 

The vast and varied ground campaign already underway turns the prediction imperative into concrete 
activity. In pursuit of economies of scope, a wave of novel machine processes are honed for extraction, 
rendering people and things as behavioral data. For the sake of economies of action, the apparatus learns 
to interrupt the flow of personal experience in order to influence, modify, and direct our behavior, guided 
by the plans and interests of self-authorizing commercial actors and the buzzing market cosmos in which 
they participate. In nearly every case the agents of institutionalization present their novel practices as if 


they are one thing, when they are, in fact, something altogether different. The realpolitik of commercial 
surveillance operations is concealed offstage while the chorus of actors singing and dancing under the 
spotlights holds our attention and sometimes even our enthusiasm. They sweat under the stage lights for 
the sake of one aim: that we fail to notice the answers or, better yet, forget to ask the questions: Who 
knows ? Who decides? Who decides who decides ? 

In light of these ambitions, it is not surprising that Doctoroff, like Page, prefers lawless space. Press 
reports confirmed that Alphabet/Google was actively considering a proposal for a new city and that more 
than a hundred urban planners, researchers, technologists, building experts, economists, and consultants 
were involved in the project. The Wall Street Journal reported that although it was unclear how the 
company would fund the tens of billions of dollars necessary for such a large-scale undertaking, “One key 
element is that Sidewalk would be seeking autonomy from many city regulations, so it could build without 
constraints....”2° 

In October 2017, Doctoroff appeared with Alphabet Executive Chairman Eric Schmidt and Canadian 
Prime Minister Justin Trudeau to reveal that Toronto would be the site of Sidewalk’s planned 
development. Its intent is to develop the right mix of technology that it can then license to cities around the 
world. “The genesis of the thinking for Sidewalk Labs came from Google’s founders getting excited 
thinking of ‘all the things you could do if someone would just give us a city and put us in charge,’” 
Toronto’s Globe and Mail reported Schmidt as saying, noting that “he joked he knew there were good 
reasons that doesn’t happen.” Then, just as quickly, the paper related Schmidt’s reaction when he first 
learned that Sidewalk, and by extension Alphabet, had secured this opportunity in Toronto: “Oh my God! 
We’ ve been selected. Now, it’s our turn.”22 


CHAPTER EIGHT 


RENDITION: FROM EXPERIENCE TO DATA 


You take a picture of ’em, they’Il kill you. 
They think you’re takin’ somethin’ away from ’em. 
That you only got so much... stuff!... and if other 
People are takin’ it all, then there ain’t none left for yourself. 


—ROBERT GARLAND, THE ELECTRIC HORSEMAN 


To photograph is to appropriate the thing photographed. 
It means putting oneself into a certain relation to the world 
that feels like knowledge—and, therefore, like power. 


—SUSAN SONTAG, ON PHOTOGRAPHY 


I. Terms of Sur-Render 


We worry about companies that amass our personal data, and we wonder why they should profit. “Who 
owns the data?” we ask. But every discussion of data protection or data ownership omits the most 
important question of all: why is our experience rendered as behavioral data in the first place? It has been 
far too easy to overlook this important step in the chain of events that produces behavioral surplus. This 
chapter and the next draw our attention to the gap between experience and data, as well as to the specific 
operations that target this gap on a mission to transform the one into the other. I call these operations 
rendition. We have seen that the dispossession of human experience is the original sin of surveillance 
capitalism, but this dispossession is not mere abstraction. Rendition describes the concrete operational 
practices through which dispossession is accomplished, as human experience is claimed as raw material 
for datafication and all that follows, from manufacturing to sales. A focus on these intermediate practices 
illustrates that the apparatus of ubiquity is not a passive one-way mirror. Rather, it actively creates its 
own stores of knowledge through rendition. 

The noun rendition derives from the verb render, a most unusual word whose double meanings 
describe a two-sided equation that perfectly captures what happens in the gap between human experience 
and behavioral data. On one side of the equation, the verb describes a process in which something is 
formed out of something else that is originally given. It designates the causal action of turning one thing 
into another, such as rendering oil from fat (extraction) or rendering an English text from the original Latin 


(translation). These meanings have also found their way into the vocabulary of digital technology. For 
example, a “rendering engine” converts the coded content of an HTML page for display and printing. 

On the other side of the equation, render also describes the way in which the thing that is changed 
gives itself over to this process: it sur-renders. The verb rendre first appears in tenth-century French, 
meaning “to give back, present, yield,” as in “rendering an account” or “the tree renders its fruit.” By the 
fourteenth century, the word also incorporated the idea of handing over, delivering, or acknowledging 
dependency or obligation, as in “Render unto Caesar.” These meanings are active today when we say 
“rendering a verdict,” “rendering service,” or “rendering property.” 

Surveillance capitalism must work both sides of the equation. On one side, its technologies are 
designed to render our experience into data, as in rendering oil from fat. This typically occurs outside of 
our awareness, let alone our consent. On the other side of the equation, every time we encounter a digital 
interface we make our experience available to “datafication,’ thus “rendering unto surveillance 
capitalism’ its continuous tithe of raw-material supplies. 

This two-sided equation is a novel arrangement. As we saw in Chapter 1, the Aware Home project 
developed at Georgia Tech just a year before the invention of surveillance capitalism employed different 
practices that embodied very different assumptions: (1) that it must be the individual alone who decides 
what experience is rendered as data, (2) that the purpose of the data is to enrich the individual’s life, and 
(3) that the individual is the sole arbiter of how the data are shared or put to use. Nearly two decades 
later, the Aware Home is barely more than an archeological fragment reminding us of the road not taken 
toward an empowering digital future and a more-just division of learning in society. Down that road, it is 
the individual who knows, decides, and decides who decides: an end in herself, not a means to others’ 
ends. The lesson of the Aware Home is that there can be rendition without surveillance capitalism. 
However, the lesson of this chapter and the next is that there can be no surveillance capitalism without 
rendition. 

Nothing is exempt, as products and services from every sector join devices like the Nest thermostat in 
the competition for surveillance revenues. For example, in July 2017 iRobot’s autonomous vacuum 
cleaner, Roomba, made headlines when the company’s CEO, Colin Angle, told Reuters about its data- 
based business strategy for the smart home, starting with a new revenue stream derived from selling floor 
plans of customers’ homes scraped from the machine’s new mapping capabilities. Angle indicated that 
iRobot could reach a deal to sell its maps to Google, Amazon, or Apple within the next two years. In 
preparation for this entry into surveillance competition, a camera, new sensors, and software had already 
been added to Roomba’s premier line, enabling new functions, including the ability to build a map while 
tracking its own location. The market had rewarded iRobot’s growth vision, sending the company’s stock 
price to $102 in June 2017 from just $35 a year earlier, translating into a market capitalization of $2.5 
billion on revenues of $660 million.4 

Privacy experts raised alarms, knowing that such data streams have virtually no legal or security 
protection. But Angle assured the public that iRobot would not sell data without its customers’ permission 
and expressed confidence that “most would give their consent in order to access the smart home 
functions.” Why was Angle so confident? 

According to the company’s privacy policy, it is true that owners of the Roomba can control or stop 
the collection of usage data by “disconnecting your WiFi or Bluetooth from the app, for example, by 
changing your WiFi password.” However, as Angle told tech site Mashable in July 2017, even when 
customers do not opt-in to the mapping service, the Roomba captures mapping and usage data, but only 
usage data “is sent to the cloud so it can be shown on your mobile device.”? What Angle neglected to say 
was that a customer who refuses to share his or her home’s interior mapping data with iRobot also loses 


most of the smart functionality of the “autonomous” vacuum cleaner, including the ability to use one’s 
phone to start or pause a cleaning, schedule cleanings, review “Clean Map reports,” receive automatic 
software updates, or start a “SPOT Clean to focus on a particularly dirty area.”4 

Angle’s confidence-enhancing strategy goes to the heart of the larger rendition project, for which 
surveillance capitalist purveyors of “smart” home products have developed a singular approach. On the 
one hand, they stress that customers can opt in to data sharing. On the other hand, customers who refuse to 
opt in face limited product functionality and data security. In these Requirimiento-style relationships, 
instead of the adelantados’ message, “Bend the knee or we destroy you,” the message here is “Bend the 
knee or we degrade your purchase.” 

Under this new regime, something as simple as buying a mattress now requires careful legal scrutiny 
of the “abusive contracts” that nearly everyone ignores. Consider the Sleep Number bed, with its “smart 
bed technology and sleep tracking.”? The company’s website features a beautiful couple snuggled in bed 
happily glued to their smartphones as they delight in data from their SleepIQ app. The bed’s base and 
mattress are “customizable” with features that raise or lower the angle of the bed and sensors that soften 
or firm up the mattress. Other sensors measure heart rate, breathing, and movement: “Every morning 
you'll get your SleepIQ® score, representing your individual quality and length of sleep... your restful 
sleep, restless sleep and time out of bed... and what adjustments you can make.” The company suggests 
that you connect your sleep app to your fitness tracker and your thermostat in order to see how your 
workout or bedroom temperature affects your sleep. 

A dense, twelve-page privacy policy accompanies the bed. Customers are advised that providing 
information is an affirmation of consent to use that information in-line with the policy, which employs the 
usual onerous terms: third-party sharing, Google analytics, targeted advertising, and much more. In 
addition, if customers create a user profile to maximize the effectiveness of the app, the company also 
collects “biometric and sleep-related data about how You, a Child, and any person that uses the Bed slept, 
such as that person’s movement, positions, respiration, and heart rate while sleeping.” It also collects all 
the audio signals in your bedroom. As with most such policies, customers are advised that the company 
can “share” or “exploit” personal information even “after You deactivate or cancel the Services and/or 
your Sleep Number account or User Profiles(s).” Customers are warned that no data transmission or 
storage “can be guaranteed to be 100% secure” and that it does not honor “Do Not Track” notifications. 
Finally, on page 8 of the document, the policy addresses a customer’s choices regarding the use of 
personal information: “Whether you submit Information to Us is entirely up to You. If you decide to not 
submit Information, We may not be able to provide certain features, products, and/or services to you.”° 

This same coercive Requirimiento twist can be found in the lengthy, dense legal compacts associated 
with Alphabet-owned Nest thermostats. The terms-of-service and end-user licensing agreements reveal 
oppressive privacy and security consequences in which sensitive information is shared with other 
devices, unnamed personnel, and third parties for the purposes of analysis and ultimately for trading in 
behavioral futures markets, action that ricochets back to the owner in the form of targeted ads and 
messages designed to push more products and services. Despite this, courts have generally upheld 
companies’ claims that they bear no liability without a clear demonstration of economic harm to the 
consumer. 

Nest takes little responsibility for the security of that information and none for how other companies 
will put it to use. In fact, University of London legal scholars Guido Noto La Diega and Ian Walden, who 
analyzed these documents, reckon that were one to enter into the Nest ecosystem of connected devices and 
apps, each with their own equally burdensome terms, the purchase of a single home thermostat entails the 
need to review nearly a thousand “contracts.” 2 


This absurdity is compounded by the fact that virtually no one reads even one such “contract.” A 
valuable empirical study of 543 participants familiar with surveillance and privacy law issues found that 
when asked to join a new online service, 74 percent opted for the “quick join” procedure, bypassing the 
terms-of-service agreement and the privacy policy. Among those who did scroll through the abusive 
contracts, most went directly to the “accept” button. The researchers calculated that the documents 
required at least forty-five minutes for adequate comprehension, but for those who looked at the 
agreements, the median time they spent was fourteen seconds.® 

Should the customer refuse to agree to Nest’s stipulations, the terms of service indicate that the 
functionality and security of the thermostat itself will be deeply compromised, no longer supported by the 
necessary updates meant to ensure its reliability and safety. The consequences can range from frozen 
pipes to failed smoke alarms to an easily hacked internal home system. In short, the effectiveness and 
safety of the product are brazenly held hostage to its owners’ submission to rendition as conquest, by and 
for others’ interests. 

One can easily choose not to purchase a Roomba, a SleepNumber bed, or a Nest thermostat, but each 
of these is merely emblematic of the immense project of rendition as the first and vital step in the 
construction of the apparatus of ubiquity. Thousands of “internet of things” objects are becoming 
available. As La Diega and Walden conclude, in this new product regime the simple product functions that 
we seek are now hopelessly enmeshed in a tangled mixture of software, services, and networks.2 

The very idea of a functional, effective, affordable product or service as a sufficient basis for 
economic exchange is dying. Where you might least expect it, products of every sort are remade by the 
new economic requirements of connection and rendition. Each is reimagined as a gateway to the new 
apparatus, praised for being “smart” while traditional alternatives are reviled for remaining “dumb.” It is 
important to acknowledge that in this context, “smart” is a euphemism for rendition: intelligence that is 
designed to render some tiny corner of lived experience as behavioral data. Each smart object is a kind of 
marionette; for all its “smartness,” it remains a hapless puppet dancing to the puppet master’s hidden 
economic imperatives. Products, services, and applications march to the drumbeat of inevitabilism 
toward the promise of surveillance revenues hacked from the still-wild spaces that we call “my reality,” 
“my home,” “my life,’ and “my body.” Every smart product repeats our essential questions: What does a 
smart product know, and whom does it tell? Who knows? Who decides? Who decides who decides? 

Examples of products determined to render, monitor, record, and communicate behavioral data 
proliferate, from smart vodka bottles to internet-enabled rectal thermometers, and quite literally 
everything in between./2 The business developer for a spirits company thus cites his plan for a “connected 
bottle”: “The more we learn about consumers and their behaviors, the better services we can connect to 
them.”!! Many brands are determined “to give packaging a speaking role in an increasingly interactive 
marketplace.” Global spirits distributor Diageo promises “smart sensor-equipped bottles” that can track 
purchases and sales data, and, most importantly, “communicate with consumers’ devices and switch gears 
—recipes versus sales promos—once the bottle is opened.” A producer of bar equipment states the case 
plainly enough: “It really is all about... allowing these [bar] owners to see stuff that they couldn’t see 
before and maximize their profits.” 2 

Today our homes are in surveillance capitalism’s crosshairs, as competitors chased a $14.7 billion 
market for smart-home devices in 2017, up from $6.8 billion just a year earlier and expected to reach 
more than $101 billion by 2021.42 You may have already encountered some of the early absurdities: smart 
toothbrushes, smart lightbulbs, smart coffee mugs, smart ovens, smart juicers, and smart utensils said to 
improve your digestion. Others are often more grim: a home security camera with facial recognition; an 
alarm system that monitors unusual vibrations before a break-in occurs; indoor GPS locators; sensors that 


attach to any object to analyze movement, temperature, and other variables; every kind of connected 
appliance; cyborg cockroaches designed to detect sound. Even the baby’s nursery is reconceived as a 
source of fresh behavioral surplus.4 

An appreciation of the surveillance logic of accumulation that drives this action suggests that this 
network of things is already evolving into a network of coercion, in which mundane functions are 
ransomed for behavioral surplus.12 A December 2013 letter from Google’s finance director to the US 
Securities and Exchange Commission’s Division of Corporate Finance provides a vivid glimpse of these 
facts. The letter was composed in response to an SEC query on the segmentation of Google’s revenues 
between its desktop and mobile platforms.t° Google answered by stating that users would be “viewing 
our ads on an increasingly wide diversity of devices in the future” and that its advertising systems were 
therefore moving toward “device agnostic” design that made segmentation irrelevant and impractical. “A 
few years from now,” the letter stated, “we and other companies could be serving ads and other content 
on refrigerators, car dashboards, thermostats, glasses, and watches, to name just a few possibilities.” 

Here is at least one endgame: the “smart home” and its “internet of things” are the canvas upon which 
the new markets in future behavior inscribe their presence and assert their demands in our most-intimate 
spaces. Key to the story is that all of this action is prosecuted in support of a larger market process that 
bets on the future of our behavior and over which we have no knowledge or control. Each node in the 
network—the vacuum cleaner, the mattress, the thermostat—must play its part, beginning with the 
frictionless rendition of behavior, as the whole team of seething insistent “smart” things joins the 
migration to surveillance revenues. As we are shorn of alternatives, we are forced to purchase products 
that we can never own while our payments fund our own surveillance and coercion. Adding insult to 
injury, data rendered by this wave of things are notoriously insecure and easily subject to breaches. 
Moreover, manufacturers have no legal responsibility to notify device owners when data are stolen or 
hacked. 

There are other, even more grandiose ambitions for the rendition of all solitary things. Companies such 
as Qualcomm, Intel, and ARM are developing tiny, always-on, low-power computer vision modules that 
can be added to any device, such as your phone or refrigerator, or any surface. A Qualcomm executive 
says that appliances and toys can know what’s going on around them: “A doll could detect when a child’s 
face turns toward it.” 

Consider “smart skin,” developed by brilliant university scientists and now poised for commercial 
elaboration. Initially valued for its ability to monitor and diagnosis health conditions from Parkinson’s 
disease to sleep disorders, smart skin is now hailed for its promise of ultra-unobtrusive ubiquity. 
Researchers at Georgia Tech developed a version of “smart skin” that sucks energy from radio waves and 
other energy sources, eliminating the need for batteries. Smart skin, described as “the ultimate sensing 
tool that could potentially allow for the mass implementation of perpetual wireless networks,”/® can 
cognize, sense, analyze, wirelessly communicate, and “modify parameters” using simple radio frequency 
(RFID) technology.“2 As in the case of Paradiso’s “sensor tape,” the researchers stress that it can also “be 
applied everywhere” to “monitor, sense, and interact with the world around us in a perpetual way, thus 
significantly enhancing ambient intelligence,” all of it as inconspicuous as a “decal sticker.” They suggest, 
for example, the shelves of grocery stores, where revenue opportunities are plentiful.2° 

Rendition has become a surveillance capitalist project shaped by its imperatives and directed toward 
its objectives. In the composition of the shadow text, rendition is Step One: the concrete 
operationalization of the “original sin of simple robbery” that defined this market project from the start. 
Google rendered the Earth, its streets, and its dwelling places, bypassing our consent and defying our 
protests. Facebook rendered the social network and its limitless details for the sake of the company’s 


behavioral futures markets. Now the ubiquitous apparatus is the means to the ubiquitous rendition of 
human experience. We have seen the urgency with which surveillance capitalists pursue the elimination of 
“friction” as a critical success factor in supply operations. The prediction imperative makes boundaries 
and borders intolerable, and surveillance capitalists will do almost anything to eliminate them. This 
pursuit transforms “connection” into a commercial imperative and transforms individual autonomy into a 
threat to surveillance revenues. 

Surveillance capitalism’s rendition practices overwhelm any sensible discussion of “opt in’ and “opt 
out.” There are no more fig leaves. The euphemisms of consent can no longer divert attention from the 
bare facts: under surveillance capitalism, rendition is typically unauthorized, unilateral, gluttonous, 
secret, and brazen. These characteristics summarize the asymmetries of power that put the “surveillance” 
in surveillance capitalism. They also highlight a harsh truth: it is difficult to be where rendition is not. As 
industries far beyond the technology sector are lured by surveillance profits, the ferocity of the race to 
find and render experience as data has turned rendition into a global project of surveillance capital. 

This chapter and the next survey a range of rendition activities in the pursuit of economies of scope. 
The remainder of this chapter concentrates on extension, the first dimension of scope, as rendition 
operations move into the real world, seizing fresh unexpected chunks of human experience. Extension 
wants every corner and crevice, every utterance and gesture on the path to dispossession. All that is moist 
and alive must hand over its facts. There can be no shadow, no darkness. The unknown is intolerable. The 
solitary is forbidden. Later, in Chapter 9, we move into the depth dimension. The net is cast wide over the 
waters of daily life, but there are also submarines exploring the depths in search of new sources of 
surplus prized for their rare predictive powers: your personality, emotions, and endorphins. The examples 
in these chapters are not intended to be exhaustive but rather to illustrate the seriousness of purpose, 
tenacity, and subterfuge with which surveillance capitalists pursue their search for new aspects of human 
experience that can be monetized as certainty. 

In this pursuit, we are necessarily involved in citing specific actors, products, and techniques, 
knowing that the details of persons and companies are constantly churning. Firms will be bought and sold, 
fail or succeed; people will come and go. Specific technologies, products, and techniques will be 
abandoned, refined, and surpassed. When they fall, new ones will take their place, as long as surveillance 
capitalism is allowed to flourish. Velocity and churn have been critical to surveillance capitalism’s 
success, and we cannot allow the constant movement to inhibit our determination to grasp the “laws of 
motion” that command this roiling landscape. It is the pattern and its purpose that we want to grasp. 


IT. Body Rendition 


The rendition of your body begins quite simply with your phone. Even when your city is not “smart” or 
owned and operated by Google, market players with an interest in your behavior know how to find your 
body.*! For all of the elaborate ways in which surveillance capitalists labor to render reality as behavior 
for surplus, the simplest and most profound is their ability to know exactly where you are all the time. 
Your body is reimagined as a behaving object to be tracked and calculated for indexing and search. Most 
smartphone apps demand access to your location even when it’s not necessary for the service they 
provide, simply because the answer to this question is so lucrative. 

Location data can be extracted from “geotags” created when your smartphone automatically embeds 
your identity and location in photos and videos. Retailers use “geofencing” to demarcate a geographical 
area and send alerts to smartphones within those parameters: “Come here now!” “Buy this here!” “An 


offer, just for you!”22 Download the Starbucks app, and then leave your house if you want to see this in 
action. As one marketing consultancy advises, “Mobile advertising, the ultimate form of geo-targeting, is 
the holy grail of advertising.”2° “Tips and tricks” for location-based marketing are generously offered by 
a firm that specializes in mobile advertising: “It allows you to tap into people’s compulsive nature by 
encouraging impulse buys with the notifications you send out.... It also allows you to gain insight on your 
current customers by reading what they’re saying on Yelp and Facebook....”24 

Another mobile marketing firm recommends “life pattern marketing” based on techniques derived from 
military intelligence known as “patterns of life analysis.” These involve gathering location and other data 
from phones, satellites, vehicles, and sensors to assemble intelligence on the daily behavioral patterns of 
a “person of interest” in order to predict future behavior. Marketers are exhorted to “map the daily 
patterns” of a “target audience” in order to “intercept people in their daily routines with brand and 
promotional messages.” As the firm emphasizes, “The psychological power of the perception of ubiquity 
is profound. Life Pattern Marketing forms a powerful psychological imprint on consumers.” 

You can shut down the GPS locator on your phone, but most people do not, both because they rely on 
its utilities and because they are ignorant of its operations. According to Pew Research, 74 percent of US 
smartphone owners in 2013 used apps that required location data, and 90 percent did so in 2015—that’s 
about 153 million people, more than those who listen to music or watch video on their phones.2° 
Surveillance capitalism’s reliance on secret operations means that most of us simply do not and cannot 
know the extent to which our phone doubles as a tracking device for corporate surveillance. 

A study by researchers at Carnegie Mellon University proves the point efficiently.2’ During a three- 
week period, twenty-three participants were continuously informed of the number of apps accessing their 
location information and the total number of accesses in a given period. They were flabbergasted by the 
sheer volume of the onslaught as they each variously learned that their locations were accessed 4,182 
times, 5,398 times, 356 times, and so on, over a 14-day period—all for the sake of advertisers, insurers, 
retailers, marketing firms, mortgage companies, and anyone else who pays to play in these behavioral 
markets.22 As one participant summed it up, “It felt like I’m being followed by my own phone. It’s 


scary.”~2 Fifty-eight percent of participants subsequently restricted the permissions granted to their mobile 


apps. 

Unsurprisingly, Google represents the vanguard of location-based tracking. A 2016 affidavit from law- 
enforcement officials seeking a search warrant for a California bank robber made plain why Google 
location data are unparalleled: “Google collects and retains location data from Android-enabled mobile 
devices. Google collects this data whenever one of their services is activated and/or whenever there is an 
event on the mobile device such as a phone call, text messages, internet access, or email access.” The 
officials on the case requested location information from Google because it offers far more detail than 
even the phone companies can provide. The location systems in Android combine cell-tower data with 
GPS, Wi-Fi networks, and other information culled from photos, videos, and other sources: “That lets 
Android pinpoint users to a single building, rather than a city block.”2? In November 2017 Quartz 
investigative reporters discovered that since early 2017, Android phones had been collecting location 
information by triangulating the nearest cell towers, even when location services were disabled, no apps 
were running, and no carrier SIM card was installed in the phone. The information was used to manage 
Google’s “push” notifications and messages sent to users on their Android phones, enabling the company 
to track “whether an individual with an Android phone or running Google apps has set foot in a specific 
store, and use that to target the advertising a user subsequently sees.”2! 

Google’s location history system is a product of the corporation’s global mapping operations. Though 
active for over a decade, it was unveiled to the public only in 2015 as “Your Timeline,” a feature that 


“allows you to visualize your real-world routines.”2* The corporation calculated that any negative 
reaction to the volume and persistence of tracking revealed by Timeline would be mitigated by the value 
of users’ active contributions to their own stocks of behavioral surplus as they fine-tune the information, 
add relevant photos, insert comments, and so forth. This was represented as an individual’s investment in 
personalized services such as Google Now so that it might more effectively comb your e-mail and apps in 
order to push relevant traffic and weather updates, notifications, suggestions, and reminders. Location 
data is the quid pro quo for these services. 

This transaction is greased by the usual promises of privacy and control: “Your Timeline is private 
and visible only to you; and you control the locations you choose to keep.” But Google uses your location 
data to target ads; indeed, these are among the most significant sources of surplus in Google’s advertising 
markets with a direct impact on click rates. The standard account from Google and other surveillance 
capitalists is that behavioral surplus is retained only as meta-data, which are then aggregated across large 
numbers of individual users. We are told that it’s not possible to identify individuals from these large- 
scale amalgamations. However, with as little as three bits of data easily culled from the public record— 
birth date, zip code, and sex—reidentification science has demonstrated its ability to de-anonymize meta- 
data with “disturbing ease.”=2 In a summary of this research, legal scholar Paul Ohm writes that “re- 
identification makes all our secrets fundamentally easier to discover and reveal. Our enemies will find it 
easier to connect us to facts that they can use to blackmail, harass, defame, frame or discriminate against 
us.... This mistake pervades nearly every information privacy law.” Regarding the massive caches of 
supposedly anonymous behavior surplus, Ohm calls them “databases of ruin.”24 

When it comes to location data, the situation is just as bad. In 2013 a group of MIT and Harvard 
computer scientists demonstrated that because individuals tend to have idiosyncratic mobility signatures, 
any analyst with the right tools can easily extract the mobility pattern of a specific individual within a 
large anonymized data set of location meta-data. Another research team demonstrated that data collected 
by seemingly “innocuous” smartphone-embedded sensors, such as accelerometers, gyroscopes, and 
magnetometers, can be used to infer “an ever-growing range of human activities and even moods.” Their 
work also shows that these sensor data can be used “to obtain sensitive information about specific users 
from anonymized datasets.” 

Companies are putting these surveillance capabilities to work. Broadcom produced a “global 
navigation satellite system” in a chip that combines satellite communications with the sensors in your cell 
phone to create a “positioning engine” that can find your location even if you are not connected to a 
network, including your location in a building, how many steps you’ve taken, in what direction, at what 
altitude. All this depends solely on one factor, says a company vice president: “the device in your 
hand.”°® Princeton computer scientists Arvind Narayanan and Edward Felten summed it up this way: 
“There is no known effective method to anonymize location data, and no evidence that it’s meaningfully 
achievable.” 

Even without “de-anonymization,” location meta-data constitute an unrivaled concentration of 
knowledge within private firms and an extraordinary advantage in the division of learning. In 2016 
Chinese search engine Baidu, often referred to as the Google of China, announced that its “Big Data Lab” 
uses location data from its 600 million users to track and predict the dynamics of the Chinese economy. 
The company built an “employment index” for the national economy as well as a “consumption index.” It 
also touted its ability to generate quite-specific predictions such as Apple’s second-quarter earnings in 
China that year. “To the best of our knowledge,” Baidu researchers wrote, “we are the first to measure the 
second-largest economy by mining such unprecedentedly large scale and fine granular spatial-temporal 
data.”38 


As powerful as location data are, wearable technologies and their applications are another significant 
proving ground in the act of body rendition.2? One 2017 report describes a new generation of wearables 
“armed with more sensors and smarter algorithms... focused on biometric monitoring... and... body parts 
as conduits for data collection....” These complex sensors can access “environmental context... smells... 
emotional state....”42 Google has developed internet-enabled fabrics, claiming that it aims to bring 
inductive yarns to every garment and fabric on Earth. “If you can weave the sensor into the textile, as a 
material,” explains project leader Ivan Poupyrev, “you’re moving away from the electronics. You’re 
making the basic materials of the world around us interactive.” A partnership with Levi Strauss has 
already yielded “interactive denim,” including a jacket first brought to market in September 2017. The 
material is described as able to “infer behavior” in order to be “interactive yet authentic.”*! The jacket 
contains sensors that can “see” through the fabric to detect and decipher gestures as subtle as the twitch of 
your finger. 

There is a numbing repetition of MacKay’s themes throughout the literature on wearables. Just as he 
insisted that telemetric devices must operate “outside the awareness” of “unrestrained animals,” today’s 
developers stress that wearables must be “unobtrusive” to avoid raising alarm. They are to be 
“continuous,” “pervasive,” and, crucially, “low cost” in order to achieve economies of scope. The 
digital marketing firm Ovum forecasts 650 million wearables by 2020, nearly double the number used in 
2016, and its research suggests that growth is largely driven by the lure of surveillance revenues. Mobile 
advertisers, they report, see wearables as “a source of very granular data insights and also new types of 
behavioral and usage data. Wearables of the future will have the ability to capture a wide array of data 
related to a user’s contextual activity, health and emotional state. This information can be used to enhance 
and tailor both products and marketing messages to a very high degree....”“ 

Health care is an especially active proving ground for wearable sensored technologies, a development 
that’s particularly pernicious given the more-innocent origins of this idea. When telemetry first shifted 
from MacKay’s gaggles, flocks, and herds to the human animal, one of its first applications was as a 
means to oversee the vulnerable in the form of push-button pendants for elderly people alone in their 
homes. In 2002, the year that a still-secret surveillance capitalism achieved its first breakthroughs, a 
review of “wireless telemedicine” stressed the value of home monitoring for the elderly and for the 
expansion of health services in remote areas. As in the case of the Aware Home, a diagram of the 
proposed digital architecture for such home-monitoring services features only three parties: a closed loop 
that exclusively links a patient at home, her hospital’s servers, and her physician.“ There are no extra 
parties imagined in either of these designs, no companies capturing your behavior, no behemoth tech firms 
with their porous platforms and proprietary servers transforming your life into surplus so they can make 
book on what you will want next and enable their customers to sell it to you first. 

Before the birth and spread of surveillance capitalism, it was possible to imagine digital 
representations of your body as an enrichment of the intimate relationships between a patient and a trusted 
doctor, a mother and her child, elderly parents and their adult children. As surveillance capitalism 
overwhelms the digital milieu, that vision has been made ridiculous. Both the Aware Home and the 
telemedicine design assume that all behavioral data are reinvested in service to the human being who is 
the subject of these arrangements, providing serenity, trust, and dignity: a chance for real knowledge and 
empowerment. 

Many articles on health monitoring continue to emphasize its utility for the elderly, but the 
conversation has decisively moved on from this earlier state of grace. Some researchers anticipate the 
fusion of “smart cities” and what’s now called “m-health” to produce “smart health,” defined as “the 
provision of health services by using the context-aware network and sensing infrastructure of smart 
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cities.”“2 Toward that end, there are now reliable sensors for rendering an increasing range of 
physiological processes as behavioral data, including body temperature, heart rate, brain activity, muscle 
motion, blood pressure, sweat rate, energy expenditure, and body and limb motion. There are sensors that 
can render audio, visual, and physiological data during postsurgical patient recovery and rehabilitation. A 
flexible, sensored textile patch has been developed that can render breathing, hand movements, 
swallowing, and walking as behavioral data. In other applications, “wearable micromachined sensors” 
provide “accurate biomechanical analysis” as you walk or run, and a “body area network” records and 
analyzes walking and running “under extreme conditions.” 

It is eloquent testimony to the health care system’s failure to serve the needs of second-modernity 
individuals that we now access health data and advice from our phones while these pocket computers 
aggressively access us. M-health has triggered an explosion of rendition and behavioral surplus capture 
as individuals turn in record numbers to their fitness bands and diet apps for support and guidance.“ By 
2016, there were more than 100,000 mobile health apps available on the Google Android and Apple iOS 
platforms, double the number in 2014.% These rich data can no longer be imagined as cloistered within 
the intimate closed loops between a patient and her doctor or between an application and its dieters or 
runners. That bucolic vision has its holdouts, to be sure, but for surveillance capitalists this vision is but a 
faded daguerreotype. 

In the US, most health and fitness applications are not subject to health privacy laws, and the laws that 
do exist do not adequately take into account either new digital capabilities or the ferocity of surveillance 
capitalist operations. Companies are expected to self-regulate by following guidelines suggested by the 
Federal Trade Commission (FTC) and other government agencies. For example, in 2016 the FTC issued a 
list of best practices for developers of mobile health applications aimed at increasing transparency, 
privacy, and security. Among these suggestions, developers are encouraged to “make sure your app 
doesn’t access consumer information it doesn’t need,” “let consumers select particular contacts, rather 
than having your app request access to all user contacts through the standard API,” and let users “choose 
privacy-protective default settings.” That year the Food and Drug Administration announced that it would 
also not seek to regulate health and fitness apps, citing their “low-level risk.” Instead, the agency released 
its own set of voluntary guidelines for software developers.” 

The agencies’ well-meaning guidelines overlook the inconvenient truth that transparency and privacy 
represent friction for surveillance capitalists in much the same way that improving working conditions, 
rejecting child labor, or shortening the working day represented friction for the early industrial capitalists. 
It took targeted laws to change working conditions back then, not suggestions. Then as now, the problems 
to which these pleas for self-restraint are addressed cannot be understood as excesses, mistakes, 
oversights, or lapses of judgment. They are necessitated by the reigning logic of accumulation and its 
relentless economic imperatives. 

A legal review of mobile health apps concludes that most of them “take the consumers’ private 
information and data without the consumers’ permission and... do not generally disclose to the user that 
this information will be sent to advertising companies.” These conclusions are borne out by a long queue 
of studies,2 but let’s focus on a 2016 in-depth investigation by scholars from the Munk School of Global 
Affairs at the University of Toronto in association with Open Effect, a nonprofit focused on digital 
privacy and security. This study looked at the collection, processing, and usage activities associated with 
nine fitness trackers.2! Seven were chosen for their popularity, one was made by a Canadian company, 
and the ninth was an app that specialized in women’s health. All but two apps transmitted every logged 
fitness event to the company’s servers, which enabled backup and sharing with one’s friends but also 
“data analytics” and distribution to third parties. Some of the trackers transmitted device identification 


numbers; others passively and continuously transmitted the user’s precise longitude and latitude 
coordinates. These identifiers “could link fitness and biographical data to a single mobile phone 
hardware, or single specific fitness wearable....” None of this sensitive information was necessary for 
the tracker to operate effectively, and most of the privacy policies were opaque at best and allowed data 
to be “sold or exchanged with third parties.” As we know, once a third party captures your surplus, it is 
shared with other third parties, who share with other third parties, and so on. 

The team also examined the trackers’ transmission of the Bluetooth Media Access Controller or 
“MAC” address that is unique to each phone. When this address is publicly discoverable, any third party 
with an interest in your movements—retailers who want to know your mall activity, insurers concerned 
about your compliance with an exercise regime—can “persistently” track your phone. Multiple data sets 
logged over time can be combined to form a fine-grained picture of your movements, enabling targeted 
applications and heightening the probability of guaranteed outcomes. The only real protection is when an 
app randomly but regularly generates a new MAC address for your phone, but of the nine trackers, only 
Apple’s performed this operation. 

The report also identifies a general pattern of careless security as well as the ability to generate false 
data. The researchers observed that consumers are likely to be misled and confused, overestimating the 
extent of security measures and underestimating “the breadth of personal data collected by fitness tracking 
companies.” As they concluded, “We discovered severe security vulnerabilities, incredibly sensitive 
geolocation transmissions that serve no apparent benefit to the end user, and... policies leaving the door 
open for the sale of users’ fitness data to third parties without express consent of the users.” 

If you are inclined to dismiss this report because fitness trackers can be written off as toys, let’s 
consider a look at an incisive investigation into Android-based diabetes apps in a 2016 Journal of 
American Medicine research report and, with it, ample illustration of the frenzy of body rendition. The 
researchers note that although the FDA approved the prescription of a range of apps that transmit sensitive 
health data, the behind-the-scenes practices of these apps are “understudied.” They examined 211 
diabetes apps and randomly sampled 65 of them for close analysis of data-transmission practices. 

Among these apps, merely downloading the software automatically “authorized collection and 
modification of sensitive information.” The researchers identified a great deal of backstage action, 
including apps that modify or delete your information (64 percent), read your phone status and identity (31 
percent), gather location data (27 percent), view your Wi-Fi connections (12 percent), and activate your 
camera in order to access your photos and videos (11 percent). Between 4 percent and 6 percent of the 
apps went even further: reading your contact lists, calling phone numbers found in your device, modifying 
your contacts, reading your call log, and activating your microphone to record your speech. 

Finally, the research team unearthed an even darker secret: privacy policies do not matter. Of the 211 
apps in the group, 81 percent did not have privacy policies, but for those that did, “not all of the 
provisions actually protected privacy.” Of those apps without privacy policies, 76 percent shared 
sensitive information with third parties, and of those with privacy policies, 79 percent shared data while 
only about half admitted doing so in their published disclosures. In other words, privacy policies are 
more aptly referred to as surveillance policies, and that is what I suggest we call them. 

There are many new territories of body rendition: organs, blood, eyes, brain waves, faces, gait, 
posture. Each of these expresses the same patterns and purpose that we have seen here. The surveillance 
capitalists relentlessly fight any attempts to constrain rendition. The ferocity with which they claim their 
“right to rendition” out of thin air is ample evidence of its foundational importance in the pursuit of 
surveillance revenues. 

This ferocity is well illustrated in surveillance capitalists’ determination to discourage, eliminate, or 


weaken any laws governing the rendition of biometric information, especially facial recognition. Because 
there is no federal law in the US that regulates facial recognition, these battles occur at the state level. 
Currently, the Illinois Biometric Privacy Act offers the most comprehensive legal protections, requiring 
companies to obtain written consent before collecting biometric information from any individual and, 
among other stipulations, granting individuals the right to sue a company for unauthorized rendition. 

The Center for Public Integrity, along with journalists, privacy advocates, and legal scholars, has 
documented the active opposition of surveillance capitalists to the Illinois law and similar legislative 
proposals in other states. With its unique competitive advantages in facial recognition, Facebook is 
considered the most uncompromising of all the tech companies when it comes to biometric data, 
described as “working feverishly to prevent other states from enacting a law like the one in Illinois.”4 

Facebook’s considerable political muscle had been cultivated in just a few years as it learned to 
emulate Google’s playbook of political and cultural fortifications. The company’s founder, Mark 
Zuckerberg, demonstrated an iron determination to preserve his freedom in lawless space, pushing the 
boundaries of existing regulations and vigorously opposing even the whisper of new law. Between 2009 
and 2017, the company increased its lobbying spend by a factor of fifty, building “a massive lobbying 
entourage of Washington power brokers.” Facebook’s $4.6 million in donations during the 2016 election 
cycle complemented its lobbying budget of $11.5 million in 2017. 

Zuckerberg’s advantages in biometrics are significant. In 2017 Facebook boasted two billion monthly 
users uploading 350 million photos every day, a supply operation that the corporation’s own researchers 
refer to as “practically infinite.”°° In 2018 a Facebook research team announced that it had “closed the 
gap” and was now able to recognize faces “in the wild” with 97.35 percent accuracy, “closely 
approaching human-level performance.” The report highlights the corporation’s supply and manufacturing 
advantages, especially the use of “deep learning” based on “large training sets.”2/ Facebook announced 
its eagerness to use facial recognition as a means to more powerful ad targeting, but even more of the 
uplift would come from the immense machine training opportunities represented by so many photos. By 
2018, its machines were learning to discern activities, interests, mood, gaze, clothing, gait, hair, body 
type, and posture.°° The marketing possibilities are infinite. 

It should not surprise any student of the prediction imperative that with these advantages in hand, 
Facebook is unwilling to accept anything less than total conquest in its bid to render faces for the sake of 
more-lucrative prediction products. So far, Facebook and its brethren have been successful, turning back 
legislative proposals in Montana, New Hampshire, Connecticut, and Alaska, and fatally weakening a bill 
that was passed in Washington state. Among the tech companies, only Facebook continued to oppose even 
the diminished terms of the Washington legislation.” 

If rendition is interrupted, surveillance capitalism cannot stand, for the whole enterprise rests on this 
original sin. This fact is amply displayed in the public drama surrounding the ill-fated 2015 attempt to 
produce public guidelines on the creation and use of biometric information through a voluntary “privacy 
multi-stakeholder” process convened by the National Telecommunications and Information Association 
(NTIA) under the auspices of the US Department of Commerce. After weeks of negotiations, consumer 
advocates walked out in protest over the hard-line position of the tech companies and their lobbyists on 
the single most pivotal issue: consent. 

The companies insisted on their right to use facial-recognition systems to identify a “stranger on the 
street” without first obtaining the individual’s consent. As one lobbyist in the talks told the press, 
“Everyone has the right to take photographs in public... if someone wants to apply facial recognition, 
should they really need to get consent in advance?” Privacy scholars were quick to respond that there is 
no lawfully established right to such actions, let alone a First Amendment right.2 Nobody reckoned with 


the fact that the prediction imperative makes individual ignorance the preferred condition for rendition 
operations, just as Arendt had observed and Mackay had prescribed for animals in the wild. Original sin 
prefers the dark. 

The talks continued without the advocates, and in 2016 the NTIA issued its “Privacy Best Practice 
Recommendations for Commercial Facial Recognition Use.” The guidelines should be understood as the 
“best” for the surveillance capitalists but as the “worst” for everyone else. In the language of these 
guidelines, the tech companies, retailers, and others determined to chase surveillance revenues are simply 
“encouraged” to make their policies on facial recognition “available to consumers, in a reasonable 
manner....” Where companies impose facial recognition on a physical location, they are “encouraged” to 
provide “notice” to consumers.®! Rendition operations are tacitly accorded legitimacy, not only in the 
lack of contest, but because they stand as the immovable facts draped in the cheap garlands of toothless 
“best practices.” Georgetown University legal scholar Alvaro Bedoya, a member of the advocacy group 
that quit the deliberations, blasted the recommendations as “a mockery of the Fair Information Practice 
Principles on which they claim to be grounded”; they offer “no real protection for individuals” and 
“cannot be taken seriously.”™ 

Under the regime of surveillance capitalism, individuals do not render their experience out of choice 
or obligation but rather out of ignorance and the dictatorship of no alternatives. The ubiquitous apparatus 
operates through coercion and stealth. Our advance into life necessarily takes us through the digital, 
where involuntary rendition has become an inescapable fact. We are left with few rights to know, or to 
decide who knows, or to decide who decides. This abnormal division of learning is created and sustained 
by secret fiat, implemented by invisible methods, and directed by companies bent to the economic 
imperatives of a strange new market form. Surveillance capitalists impose their will backstage, while the 
actors perform the stylized lullabies of disclosure and agreement for the public. 

The prediction imperative transforms the things that we have into things that have us in order that it 
might render the range and richness of our world, our homes, and our bodies as behaving objects for its 
calculations and fabrications on the path to profit. The chronicles of rendition do not end here, however. 
Act II requires a journey from our living rooms and streets to another world below the surface, where 
inner life unfolds. 


CHAPTER NINE 


RENDITION FROM THE DEPTHS 


I couldn't feel, so I tried to touch... 


—LEONARD COHEN 
“HALLELUJAH” 


I. Personalization as Conquest 


Microsoft CEO Satya Nadella introduced Cortana, the corporation’s “personal digital assistant,” at the 
firm’s annual Ignite conference in 2016: 


This new category of the personal digital assistant is a runtime, a new interface. It can take text input. It can take speech input. It 
knows you deeply. It knows your context, your family, your work. It knows the world. It is unbounded. In other words, it’s about you; 


it’s not about any one device. It goes wherever you go. It’s available on any phone—iOS, Android, Windows—doesn’t matter. It is 


available across all of the applications that you will use in your life.4 


This is a new frontier of behavioral surplus where the dark data continent of your inner life—your 
intentions and motives, meanings and needs, preferences and desires, moods and emotions, personality 
and disposition, truth telling or deceit—is summoned into the light for others’ profit. The point is not to 
cure but to render all of it as immeasurably tiny bits of behavior available for calculation so that each can 
take its place on the assembly line that moves from raw materials to product development, manufacturing, 
and sales. 

The machine invasion of human depth is prosecuted under the banner of “personalization,” a slogan 
that betrays the zest and cynicism brought to the grimy challenge of exploiting second-modernity needs 
and insecurities for outsize gain. From the point of view of the prediction imperative, personalization is a 
means of “individualizing” supply operations in order to secure a continuous flow of behavioral surplus 
from the depths. This process can be accomplished successfully only in the presence of our unrelenting 
hunger for recognition, appreciation, and most of all, support. 

Recall that Hal Varian, Google’s chief economist, helped chart this course. “Personalization and 
customization” are the third “new use” of computer-mediated transactions. Instead of having to ask 
Google questions, it should “know what you want and tell you before you ask the question.” Google Now, 
the corporation’s first digital assistant, was charged with this task. Varian warned that people would have 


to give Google even more of themselves in order to reap the value of the application: “Google Now has 
to know a lot about you and your environment to provide these services. This worries some people.” He 
rationalizes any concern, arguing that rendering personal information to Google is no different from 
sharing intimacies with doctors, lawyers, and accountants. “Why am I willing to share all this private 
information?” he asks. “Because I get something in return.... These digital assistants will be so useful that 
everyone will want one.” Varian is confident that the needs of second-modernity individuals will subvert 
any resistance to the rendition of personal experience as the quid pro quo for the promise of a less 
stressful and more effective life.? 

In fact, Varian’s notion of personalization is the precise opposite of the relationships with trusted 
professionals to which he refers. Doctors, accountants, and attorneys are held to account by mutual 
dependencies and reciprocities dictated by the extensive institutionalization of professional education, 
codes of conduct, and procedures for evaluation and review. Violation of these rules risks punishment in 
the form of professional sanction and public law. Google and its brethren in surveillance capitalism bear 
no such risks. 

Varian’s remarks constitute one of those rare occasions in which the fog of technology rhetoric parts 
just enough to discern the utilities of social and economic inequality to surveillance capitalism’s larger 
mission. Varian reasons that inequality offers an opportunity to raise the ante on Google’s quid pro quo for 
effective life. He counsels that the way to predict the future is to observe what rich people have because 
that’s also what the middle class and the poor will want. “What do rich people have now?” he asks 
rhetorically. “Personal assistants.” 

That the luxuries of one generation or class become the necessities of the next has been fundamental to 
the evolution of capitalism during the last five hundred years. Historians describe the “consumer boom” 
that ignited the first industrial revolution in late-eighteenth-century Britain, when, thanks to visionaries 
like Josiah Wedgewood and the innovations of the early modern factory, families new to the middle class 
began to buy the china, furniture, and textiles that only the rich had previously enjoyed. This new 
“propensity to consume” is considered “unprecedented in the depth to which it penetrated the lower 
reaches of society....”2 In 1767 the political economist Nathaniel Forster worried that “fashionable 
luxury” was spreading “like a contagion,” and he complained of the “perpetual restless ambition in each 
of the inferior ranks to raise themselves to the level of those immediately above them.”4 Adam Smith 
wrote insightfully on this social process, noting that upper-class luxuries can in time be recast as 
“necessaries.” This occurs as “the established rules of decency” change to reflect new customs 
introduced by elites, triggering lower-cost production methods that transform what was once unattainable 
into newly affordable goods and services.2 Ford’s Model T is the outstanding twentieth-century example 
of this progression. 

Varian casts personalization as a twenty-first-century equivalent of these historical dynamics, the new 
“necessaries” for the harried masses bent under the weight of stagnant wages, dual-career obligations, 
indifferent corporations, and austerity’s hollowed-out public institutions. Varian’s bet is that the digital 
assistant will be so vital a resource in the struggle for effective life that ordinary people will accede to its 
substantial forfeitures. “There is no putting the genie back in the bottle,” Varian the inevitabilist insists. 
“Everyone will expect to be tracked and monitored, since the advantages, in terms of convenience, safety, 
and services, will be so great... continuous monitoring will be the norm.”® Everyone, that is, except those 
wealthy or stubborn enough to achieve effective life without Google’s assistance and thus escape the 
worst excesses of rendition. As decision rights and self-determination become privileges of the wealthy, 
what will Varian offer to answer those who clamor for the same? 

Historically, breakthroughs in lower-cost goods and services unleashed expansions of production and 


employment, higher wages, and an improved standard of living for many. Varian has no such reciprocities 
in mind. Instead, he pokes his finger into the open wound of second-modernity insecurities and bends our 
pain to the objectives of the surveillance project. With Varian, the hunger for new necessities is construed 
as an opportunity to dispossess, even as it conveniently provides the justification for that dispossession 
right down to the depths. 

Google Now was a first step, although later it would look more like a stalking horse and habituation 
exercise paving the way for what was to come. Christened “predictive search,” it combined every system 
that Google had ever built, including the corporation’s achievements in voice search and neural 
networking, its knowledge of the world represented in its one-billion-entity “knowledge graph,” and its 
unparalleled machine intelligence capabilities. All of this firepower was amassed in order to learn from 
your content, context, and behavior not only through search, e-mail, and calendar activity but also from the 
data in your phone, including movement, location, activities, voice, and apps. This time, the aim was not 
only to sell ads but rather “to guess the information you’ ll need at any given moment” as you move through 
the real world.2 

As a promotional video crows, “Google Now is always one step ahead so you can feel more confident 
as you navigate your day... with the predictive power of Now, you get just what you need to know, right 
when you need it.” One writer described the new service as “having the search engine come to you.”® The 
app’s information cards swim into view on your phone’s home screen in anticipation of your needs: 
notification of a changed flight time, impending weather and traffic, nearby restaurants and shops, that 
museum you’ve been wanting to visit. One Google executive reasoned that Google already knows all of 
this about you, so it might as well turn it into a service that can provide the company with access to even 
more information: “Google’s going to know when my flight is, whether my package has gotten here yet, 
and where my wife is and how long it’s going to take her to get home this afternoon.... Of course Google 
knows that stuff.”2 Google Now’s predictive capabilities follow the pattern we have seen throughout: they 
derive from machine processes trained on unceasing flows of virtual and real-world behavior. Why did 
Google devote so much machine power and valuable surplus in order to thoughtfully assist you through 
your day? The reason is that Google Now signaled a new breed of prediction products. 

Google’s breakthrough crawler enabled the lightning-fast indexing of the world wide web, the 
apparatus of ubiquity then enabled new operations to crawl reality, and now in this third phase, distinct 
supply operations are required to crawl our lives. In Google Now one sees an initial foray into this new 
space, in which the web crawler’s ability to find information combines with new life-crawling 
operations intended to render, anticipate, and, as we shall see, ultimately modify your behavior. Online 
and offline behavioral surplus—your e-mail content, where you went this afternoon, what you said, what 
you did, how you felt—are combined into prediction products that can serve an emerging marketplace in 
which every aspect of your daily reality is up for bid. 

Facebook’s “M,” launched in 2015 as part of its Messenger application, is another example of this 
new phase. It was introduced as a “personal digital assistant... that completes tasks and finds information 
on your behalf... powered by artificial intelligence that’s trained and supervised by people.”22 
Facebook’s vice president in charge of messaging products described the company’s goals for M by 
saying, “We start capturing all of your intent from the things you want to do. Intent often leads to buying 
something, or to a transaction and that’s an opportunity for us to [make money] over time.” Most 
importantly, the VP stressed, “M learns from human behaviors.”!! The corporation’s machines would be 
trained on surplus from Messenger’s 700 million daily users. Eventually, it was hoped, M’s operations 
would be fully automated and would not need human trainers. 

By 2017, Facebook had scaled back its machine intelligence ambitions and focused its personal 


assistant on the core mission: commerce. “The team in there now is finding ways to activate commercial 
intent inside Messenger,” a Facebook executive reported.“ The idea is to “prioritize commerce-driven 
experiences” and design new ways for users to “quickly buy things” without the tedium of entering credit 
card information, flipping pages, or opening applications. Pop-up buttons appear during your 
conversations with friends whenever the system detects a possible “commercial intention.” Just tap to 
order, buy, or book, and let the system do the rest.42 

In this way the “personal digital assistant” is revealed as a market avatar, another Trojan horse in 
which the determination to render and monetize your life is secreted under the veil of “assistance” and 
embellished with the poetry of “personalization.” Its friendly recommendations, advice, and eagerness to 
act on your behalf barely conceal an aggressive new market cosmos hovering over any and every aspect 
of your daily life. It may be composed of restaurants, banks, plumbers, merchants, ticket sellers, airlines, 
and a limitless queue of possible strangers summoned by their interests in your behavior: now, soon, and 
later. They are standing by to cash in on your walk to work, your conversation with your teenager, or your 
aging running shoes. A digital assistant may derive its character from your inclinations and preferences, 
but it will be skewed and disfigured in unknown measure by the hidden market methods and contests that 
it conceals. 

Google joined other tech companies determined to establish “conversation” as the medium through 
which humans engage with the apparatus. In time, the obsession with voice may be surpassed or joined by 
others so that merely thinking a thought or waving a finger can translate into and initiate action. For now, 
there are compelling reasons for the race to the spoken word. The first is obvious: reliable voice 
recognition can translate a sprawling landscape of service interactions into low-cost automated processes 
of theoretically unlimited scale and scope, a fact that has been noted by labor economists for some time.14 
The competitive race between a new crop of “personal digital assistants” is best understood from this 
point of view. The voice that rises to dominance, the One Voice, will be the colossus of behavioral 
surplus pipelines with a potentially insurmountable competitive advantage in its ability to corner and 
kidnap the dominant share of human experience. 

“Conversation” stands alone in its promise to dominate raw-material supply, and the rewards to the 
One Voice would be astronomical. Casual talk helps to blur the boundaries between “it”—the apparatus 
saturated with commercial agents—and us. In conversation we imagine friendship. The more we fancy the 
apparatus as our confidante, nanny, governess, and support system—a disembodied, pervasive “Mrs. 
Doubtfire” for each person—the more experience we allow it to render, and the richer its supply 
operations grow. Communication is the first human joy, and a conversational interface is prized for the 
frictionless ease in which a mere utterance can trigger action, especially market action: “Let there be 
light.” “Let there be new running shoes.” What could be dreamier than to speak and have it be so? An 
Amazon senior vice president comments on the company’s voice-activated home devices: “The nice thing 
about the Amazon device business is that when we sell a device, generally people buy more blue jeans. 
And little black dresses. And shoes. And so that’s good.” “Voice shopping,” he concludes, is good for 
business and good for predicting business. 

In a conversation addressed to a digital thing, as opposed to a conversation in a shop, words can occur 
on the fly with less friction and effort; less inhibition, fretting, and comparing; less concern about the 
limits of one’s bank account or where a product or service is sourced; less doubt and hesitation; less 
memory and remorse. The speaker feels herself at the center of a seamlessly flowing universe. The seams 
are all backstage, where the machines confront and conquer stubborn sources of friction such as distinct 
apps and entities; recalcitrant administrative service, distribution, payments, and delivery systems; and 
boundaries and borders that threaten the flows of desire and satisfaction. Spontaneous and fluid, 


universally burbling “conversation” turns the new personal digital assistant into a voice that sits between 
your life and the new markets for your life, between your experience and the auctioning of your 
experience: “a runtime, a new interface” that creates the sensation of mastery while, in fact, giving it 
away. 

In this commercial dreamscape, words that were once conceived of as “behind closed doors” are 
eagerly rendered as surplus. These new supply operations convert your talk into behavior for surplus in 
two ways. The first derives from what you say, the second from how you say it. Smart-home devices such 
as Amazon’s Echo or Google Home render rivers of casual talk from which sophisticated content 
analyses produce enhanced predictions that “anticipate” your needs. Google used its 2016 developers 
conference to introduce its conversational reimagining of Google Now, rechristened “Assistant” and 
integrated across the company’s devices, services, tools, and applications. “We want users to have an 
ongoing, two-way dialogue with Google. We want to help you get things done in your real world and we 
want to do it for you,” explained Google CEO Sundar Pichai. “For example, you can be in front of this 
structure in Chicago and ask Google, ‘Who Designed This?’ You don’t need to say ‘the bean’ or ‘the 
cloud gate.’ We understand your context and we answer that the designer is Anish Kapoor.” © 

Google’s Assistant arrived already integrated into Google’s new messaging app, Allo, where it can 
search for information, execute tasks, or even compose routine messages on your behalf. Most 
importantly, Assistant animates the firm’s home device, Google Home.’ The idea is that in time, the 
device (or its successor) will claim for rendition a theoretically limitless scope of animate and inanimate 
domestic activities: conversations, lightbulbs, queries, schedules, movement, travel planning, heating 
systems, purchases, home security, health concerns, music, communication functions, and more. 

There was a time when you searched Google, but now Google searches you. Advertisements for 
Google Home feature loving families leading busy, intricate lives but visibly relieved to return home and 
fall into the arms of this omniscient, efficient caretaker. This second-modernity dream come true extracts 
an unusually high tax for its promise of a more effective life. For each user to have his or her own 
individual Google, as Pichai envisions, Google must have each individual .18 

The caretaker’s ability to effectively serve you depends entirely upon the degree to which your life is 
rendered, knowingly or unknowingly, to its ministrations. The breadth and depth of rendered life 
correspond to the scale of market action that can be triggered and mediated by Assistant. There are 
differences among the various incarnations of “personalization” and “assistance” offered by the tech 
giants, but these are trivial compared with the collective urge toward total knowledge—about your inner 
states, real-world context, and specific daily life activities—all in the service of successfully training the 
machines that they might better target market operations to each moment of life. 

All the potential market action associated with what you say depends upon voice activation, 
recognition, and response. These, in turn, are the product of highly sophisticated machine systems trained 
on vast global stockpiles of spoken words. The more structural insights the machines glean from spoken 
surplus, the more commerce flows from its content. This means that the value of what you say cannot be 
realized without machines that can learn from precisely how you say it. This form of surplus derives from 
the structure of your speech: vocabulary, pronunciation, intonation, cadence, inflection, dialect. 

The competition for supplies of talk turns your phrases into this second form of surplus as companies 
determined to develop and perfect voice capabilities scour the world for speech. “Amazon, Apple, 
Microsoft and China’s Baidu have embarked on a worldwide hunt for terabytes of human speech,” reports 
Bloomberg Businessweek. “Microsoft has set up mock apartments in cities around the globe to record 
volunteers speaking in a home setting.” The tech firms capture flows of talk from their smart devices and 
phones as they record and retain your words. Chinese search firm Baidu collects speech in every dialect: 


“Then they take all that data and use it to teach their computers how to parse, understand, and respond to 
commands and queries.” 12 

Pieces of your talk are regularly farmed out in bulk to third-party firms that conduct “audio review 
processes” in which virtual scorers, tasked to evaluate the degree of match between the machine’s text 
and the original chunk of human speech, review audio recordings retained from smartphones, messaging 
apps, and digital assistants. Companies such as Amazon, Google, and Microsoft use these audio analyses 
to improve the algorithms of their voice systems. The tech companies insist that such recordings are 
anonymous, nothing more than voices without identities. “Partners do not have the ability to tie the voice 
samples back to specific people,” one Microsoft executive asserted. But one journalist who signed on to a 
virtual job as an audio recording analyst concluded just the opposite, as she listened to recordings full of 
pathos, intimacy, and easily identifiable personal information: 


Within the recordings themselves, users willingly surrender personal information—information that is especially valuable in these 
review processes because they are so specific. Uncommon names, difficult-to-pronounce cities and towns, hyperlocal oddities.... I 
heard people share their full names to initiate a call or offer up location-sensitive information while scheduling a doctor’s 


appointment... the recordings capture people saying things they’d never want heard, regardless of anonymity.... There isn’t much to 


keep people who are listening to these recordings from sharing them.22 


There is also substantial capital investment directed at talk, and Samsung’s Smart TV illustrates some 
of this behind-the-scenes action. Business forecasts routinely predict strong growth in the market for 
internet-enabled appliances, and Samsung is among a small group of market leaders. Its appliances use 
the Android operating system platform, and early on the firm established alliances with both the 
Alphabet/Google subsidiary Nest and with Cisco. “Our first mission is to bring your home to your 
connected life,” a top executive explained in 2014.*! In 2015 privacy advocates discovered that the 
corporation’s smart TVs were actually too smart, recording everything said in the vicinity of the TV— 
please pass the salt; we’re out of laundry detergent; I’m pregnant; let’s buy a new car; we’re going to 
the movies now; I have a rare disease; she wants a divorce; he needs a new lunch box; do you love 
me ?—and sending all that talk to be transcribed by another market leader in voice-recognition systems, 
Nuance Communications.” 

The TV’s “surveillance policy’—yes, even a TV has a surveillance policy now—reveals the layers 
of surveillance effort and commercial interest that operate outside of awareness in our homes. Samsung 
acknowledges that the voice commands aimed at triggering the TV’s voice-recognition capabilities are 
sent to a third party and adds, “Please be aware that if your spoken words include personal or other 
sensitive information, that information will be among the data captured and transmitted to a third party 
through your use of Voice Recognition.”22 Samsung disclaims responsibility for the policies of third-party 
firms, as nearly all surveillance policies do, including the one that actually collects and translates the talk 
of its unsuspecting customers. Samsung advises that “You should exercise caution and review the privacy 
statements applicable to the third-party websites and services you use.” The intrepid consumer 
determined to study these documents will find no succor in Nuance’s privacy policy, only the same 
catechism found in Samsung’s and that of nearly every company. It also encourages you to read the 
privacy policies of the companies to whom it’s selling your conversations, and so it goes: a forced march 
toward madness or surrender.” 

In California, at least, the legislature passed a law prohibiting connected TVs from collecting voice 
data without “prominently informing” customers and further outlawed the use of such data for third-party 
advertising.*° As we know from our examination of the dispossession cycle, however, the economic 


imperatives that drive surveillance capitalists to capture behavioral surplus are not easily deterred. By 
2016, Samsung had doubled down on its smart-TV—based secret rendition and behavioral surplus supply 
chains, positioning its new models as the hub of a new “Samsung SmartThings smart-home ecosystem in 
an open platform that supports thousands of devices,” including fans, lights, thermostats, security cameras, 
and locks—and all with the aid of a single universal remote able to capture your every spoken 
command. 

In 2017 the FTC reached a $2.2 million settlement of a complaint initiated by the Office of the New 
Jersey Attorney General against Vizio, one of the world’s largest manufacturers and distributors of 
internet-enabled smart televisions. Vizio’s supply operations appear to be even more aggressive than 
Samsung’s. Investigators discovered that “on a second-by-second basis, Vizio collected a selection of 
pixels on the screen that it matched to a database of TV, movie, and commercial content.” The company 
then identified additional viewing data “from cable or broadband service providers, set-top boxes, 
streaming devices, DVD players, and over-the-air broadcasts.” All of this amounted to as much as 100 
billion data points each day just from the 11 million TVs identified in the settlement.28 Vizio disguised its 
supply operations behind a setting called “Smart Interactivity,” described to consumers as a feature that 
“enables program offers and suggestions” without any indication of its actual functions. 

In an unusually vivid blog post, the FTC describes Vizio’s direct sales of this behavioral surplus: 


Vizio then turned that mountain of data into cash by selling consumers’ viewing histories to advertisers and others. And let’s be clear: 
We’re not talking about summary information about national viewing trends. According to the complaint, Vizio got personal. The 
company provided consumers’ IP addresses to data aggregators, who then matched the address with an individual consumer or 
household. Vizio’s contracts with third parties prohibited the re-identification of consumers and households by name, but allowed a 
host of other personal details—for example, sex, age, income, marital status, household size, education, and home ownership. And 


Vizio permitted these companies to track and target its consumers across devices.22 


A concurring statement from FTC Acting Chairwoman Maureen K. Ohlhausen emphasized that the 
settlement broke new ground in its allegation that “individualized television viewing activity falls within 
the definition of sensitive information” that merits protection by the FTC.’ This finger in the dike would 
not hold back the tidal wave of similar incursions as the prediction imperative cracks the whip to drive 
the hunt for unexplored pieces of talk from daily life. Rendition takes command of even the most benign 
supply sources, such as toys, which have now become “toys that spy.” A new breed of interactive dolls 
and toy robots, including a girl doll called “My Friend Cayla,” turn out to be supply hubs for underage 
behavioral surplus, subjecting young children and their parents’ smartphones “to ongoing surveillance... 
without any meaningful data protection standards.” 

The popular playthings, marketed by Genesis Toys, are bundled with a mobile application that, once 
downloaded to a smartphone, “provides the data processing” to enable the toy’s ability to capture and 
understand whatever the child says.2* Along the way, the app accesses most of the phone’s functions, 
including many that are irrelevant to the toy’s operations such as contact lists and the camera. The app 
establishes a Bluetooth connection that links the toy to the internet, and it records and uploads 
conversations as the toy actively engages the child in discussion. One doll targeted in the complaint 
systematically prompts children to submit a range of personal information, including where they live. 

The child’s conversations are translated into text by third-party voice-recognition software from, once 
again, Nuance Communications, and that information is used to retrieve answers to the child’s questions 
using Google Search and other web sources. Researchers discovered that the audio files of children’s talk 
(Nuance calls them “dialogue chunks”) are uploaded to the company’s servers, where they are analyzed 


and stored.*2 As you might expect, those dialogue chunks continue their journey as behavioral surplus, in 
much the same way as Samsung’s TV-captured audio, to be sold again and again “for other services and 
products,” as the Genesis terms-of-service agreement indicates. 

Meanwhile, Mattel, one of the world’s largest toy companies, was gaining ground with its innovations 
in interactive, internet-enabled, machine-intelligence-powered toys, led by the new conversational 
Barbie Doll and its Barbie Dream House.“ The voice-activated smart dollhouse could respond to more 
than one hundred commands, such as “lower the elevator” and “turn on the disco ball,” a new kind of 
habituation exercise aimed at normalizing ubiquity in intimate spaces. “Barbie’s New Smart Home Is 
Crushing It So Hard,” exulted Wired. “Barbie’s ultimate crib is voice controlled.... It’s what a real smart 
home should be: Straight crushing it with universal voice control instead of a hodgepodge of disparate 
appliances hogging up app space on your phone.... The future is served.”2 

In this future, children learn the principles of the One Voice—a run time, a new interface. It is 
available everywhere to execute their commands, anticipate their desires, and shape their possibilities. 
The omnipresence of the One Voice, with its fractious, eager marketplace-of-you concealed under its 
skirts, changes many things. Intimacy as we have known it is compromised, if not eliminated. Solitude is 
deleted. The children will learn first that there are no boundaries between self and market. Later they will 
wonder how it could ever have been different. 

When Mattel hired a new CEO in January 2017, it’s no surprise that the company looked to Google, 
selecting its president for the Americas with responsibility for Google’s commercial and advertising 
sales operations.2° Most analysts agreed that the appointment heralded Mattel’s commitment to its 
innovations in internet-enabled toys and virtual reality, but the appointment underscores the shift in focus 
from making great products for you to collecting great data about you. 

The doll that was once a beloved mirror of a child’s unfettered imagination, along with all the other 
toys in the toy box—and the box, and the room that hosts the box, and the house that hosts the room—are 
all earmarked for rendition, calculation, connection, and profit. No longer mere things, they are reinvented 
as vehicles for a horde of commercial opportunities fabricated from our dialogue chunks and assorted 
gold dust. 

In 2017 Germany’s Federal Network Agency banned the Cayla doll as an illegal surveillance device 
and urged parents to destroy any such dolls in their possession. In the US the FTC had yet to take any 
action against the doll or Genesis Toys. Meanwhile, the connected dollhouse prepares our children and 
families for the connected room (a project that Mattel announced in January 2017 and then nine months 
later shelved amid an uproar from parents and privacy advocates), which paves the way for the connected 
home, whose purveyors hope will numb us to the connected world as we travel the path of ubiquity’s 
manifest destiny and its promise of surveillance revenues.“ 

In pursuit of the what and the how of voice surplus, the logic of competition is to corner as much 
supply as possible. The urge toward totality generates competitive pressures to become the run time and 
the new interface: the dominant, if not exclusive, medium through which we access and engage the 
apparatus as it engages us. It’s a race to corner all the talk as a prerequisite for achieving the privileged 
status of the One Voice, which will bestow upon the winner the ability to anticipate and monetize all the 
moments of all the people during all the days. 

The messianic urge toward totality and supremacy is evident in the rhetoric and strategies of key 
competitors in this race. Although Google, Microsoft, Amazon, and Samsung each have aspirations to 
dominate voice capture, it is Amazon, its machine learning assistant Alexa, and its expanding line of Echo 
hubs and Dot speakers that offer the most telling case here. Alexa appears to be a threshold event that will 
define Amazon not only as an aggressive capitalist but also as a surveillance capitalist.*° 


Amazon aggressively opened Alexa to third-party developers in order to expand the assistant’s range 
of “skills,” such as reading a recipe or ordering a pizza. It also opened its platform to smart-home device 
makers from lighting systems to dishwashers, turning Alexa into a single voice for controlling your home 
systems and appliances. In 2015 Amazon announced that Alexa would be sold as a service, known as 
“Amazon Lex,” enabling any company to integrate Alexa’s brain into its products. Amazon Lex is 
described as “a service for building conversational interfaces into any application using voice and text.... 
Lex enables you to define entirely new categories of products.”22 As Alexa’s senior vice president 
explained, “Our goal is to try to create a kind of open, neutral ecosystem for Alexa... and make it as 
pervasive as we possibly can.” 

By 2018, Amazon had inked deals with home builders, installing its Dot speakers directly into ceilings 
throughout the house as well as Echo devices and Alexa-powered door locks, light switches, security 
systems, door bells, and thermostats. As one report put it, “Amazon can acquire more comprehensive data 
on people’s living habits....” The company wants to sell real-world services such as house cleaning, 
plumbing, and restaurant delivery, but according to some insiders, the vision is more far-reaching: an 
omniscient voice that knows all experience and anticipates all action.4! Already, forward-looking 
Amazon patents include the development of a “voice-sniffer algorithm” integrated into any device and 
able to respond to hot words such as “bought,” “dislike,” or “love” with product and service offers. 

Amazon is on the hunt for behavioral surplus.“? This explains why the company joined Apple and 
Google in the contest for your car’s dashboard, forging alliances with Ford and BMW. “Shopping from 
the steering wheel” means hosting behavioral futures markets in your front seat. Alexa is ready with 
restaurant recommendations or advice on where to get your tires checked. “As pervasive as possible” 
explains why Amazon wants its Echo/Alexa device to also function as a home phone, able to make and 
receive calls; why it inked an agreement to install Echo in the nearly 5,000 rooms of the Wynn resort in 
Las Vegas; and why it is selling Alexa to call centers to automate the process of responding to live 
questions from customers by phone and text.“ Each expansion of Alexa’s territory increases the volume 
of voice surplus accrued to Amazon’s servers and fed to Alexa. 

The path to the coronation of the One Voice is not an easy one, and there are other travelers determined 
to elbow their way to the finish line. Google also wants its “personal assistant,” Google Home, to double 
as your home phone. Samsung reappears as another contender with its acquisition of “Viv,” a powerful 
new voice system designed by the original developers of Apple’s personal assistant Siri, who were 
frustrated with the constraints of Apple’s approach. Viv’s lead developer explained that “you can get 
things done by talking to things... a marketplace that will become the next big area...” 

If life is a wild horse, then the digital assistant is one more means by which that horse is to be broken 
by rendition. Unruly life is brought to heel, rendered as behavioral data and reimagined as a territory for 
browsing, searching, knowing, and modifying. Just as surveillance capitalism transformed the web into a 
market onslaught fueled by the capture and analysis of behavioral surplus, so everyday life is set to 
become a mere canvas for the explosion of a new always-on market cosmos dedicated to our behavior 
and from which there is no escape. 


IT. Rendition of the Self 


“We are used to face-to-face interaction where words disappear.... I assumed that keyboard 
communication was like a letter or a phone call, but now I understand that it doesn’t disappear. The myth 
is that electronic communication is invisible....”“° He was a brilliant research scientist at a large 


pharmaceuticals company that I called “Drug Corp” in my 1988 book, In the Age of the Smart Machine.” 
I had spent several years visiting the research group there as they shifted much of their daily 
communications from face-to-face meetings to DIALOG, one of the world’s first “computer conferencing” 
systems. DIALOG was a precursor to a technology that we now call “social media.” The DIALOG 
platform created a new social space in which the scientific community at Drug Corp elaborated and 
extended “their networks of relationships, access to information, thoughtful dialogue, and social banter,” 
as I described it then. They embraced DIALOG with great enthusiasm, but it did not end well. “In time,” I 
wrote, “it became clear that they had also unwittingly exposed once evanescent and intangible aspects of 
their social exchange to an unprecedented degree of hierarchical scrutiny.” The interviews that stretched 
across those years documented the scientists’ gradual awakening to new dangers as dimensions of 
personal experience that were implicit and private suddenly became explicit and public in ways that they 
did not anticipate and with consequences that they lamented deeply. 

Thanks to the new computer-mediated milieu, the scientists’ social and professional conversations 
now appeared as an electronic text: visible, knowable, shareable. It enriched their work in many ways, 
but it also created unexpected vulnerabilities as dispositions, values, attitudes, and social interactions 
were recast as objects of scrutiny. In a series of conflicts that unfolded over the years, I watched as the 
managers and executives at Drug Corp were simply unable to quell their inclination to use the new social 
text aS a means to evaluate, critique, and punish. On more than one occasion I learned of managers who 
had printed out pages of DIALOG conversations in order to spread them out on the floor and analyze 
opinions on a particular subject, sometimes actually taking scissors to the pages and organizing the entries 
by theme or person. In many cases these investigations were pure fact gathering, but in other instances 
managers wanted to identify the individuals who agreed with or opposed their directives. 

The enduring witness of the text was adapted as a medium through which managers attempted “to 
control and channel what had always been the most ephemeral aspects of subordinates’ behavior.” All 
the original excitement and promise melted into cynicism and anxiety as the scientists gradually withdrew 
from DIALOG, opting instead for a routine e-mail application and a preference for more-perfunctory, 
impersonal messages. 

Decades later, the children and grandchildren of those scientists, along with most of us, communicate 
freely through our smartphones and social media, unaware that we are reliving the bitter lessons of Drug 
Corp but now at an entirely new level of rendition. The scientists were rattled to see their casual talk 
reified and converted into an object of hierarchical inspection. Now, the interiors of our lives— 
simplistically summarized as “personality” or “emotions”’—are recast as raw material available to 
anyone who can make or buy a new generation of supply-chain accessories and the means of production to 
analyze this new genus of behavioral surplus and fabricate extremely lucrative prediction products. 

“Personalization” is once again the euphemism that spearheads this generation of prediction products 
manufactured from the raw materials of the self. These innovations extend the logic of earlier iterations of 
dispossession: from web crawling to reality crawling to life crawling to self crawling. As has been the 
case in each iteration, insights and techniques once intended to illuminate and enrich quickly disappear 
into the magnetic field of the commercial surveillance project, only to reappear later as ever more 
cunning methods of supply, manufacture, and sales. 

The two billion and counting Facebook users are the most poignant descendants of Drug Corp’s 
scientists. Many of them joined Facebook to escape the pervasive hierarchical oversight of workplace 
communications that had become routine in the years since Drug Corp’s first experiments. Facebook, they 
once thought, was “our place,” as benign and taken for granted and as the old Ma Bell, a necessary utility 
for association, communication, and participation. Instead, Facebook became one of the most authoritative 


and threatening sources of predictive behavioral surplus from the depths. With a new generation of 
research tools it learned to plunder your “self” right through to your most intimate core. New supply 
operations can render as measurable behavior everything from the nuances of your personality to your 
sense of time, sexual orientation, intelligence, and scores of other personal characteristics. The 
corporation’s immense machine intelligence capabilities transform these data into vivid prediction 
products. 

The stage was set for the discovery of your Facebook profile’s easy pillage in 2010, when a 
collaboration of German and US scholars came to the unexpected conclusion that Facebook profiles are 
not idealized self-portraits, as many had assumed. Instead, they found that the information on Facebook 
reflects the user’s actual personality, as independently assessed by the well-validated protocols of the 
five-factor personality model and as compared to study participants’ own descriptions of their “ideal 
self.” 

There is compelling evidence to suggest that the unique dynamics of the Facebook milieu eventually 
complicated this picture of “real personality,” as we shall explore in Chapter 16, but in 2011 these early 
findings encouraged three University of Maryland researchers to take the next logical step. They 
developed a method that relies on sophisticated analytics and machine intelligence to accurately predict a 
user’s personality from publicly available information in his or her Facebook profile. 

In the course of this research, the team came to appreciate the magic of behavioral surplus, 
discovering, for example, that a person’s disclosure of specific personal information such as religion or 
political affiliation contributes less to a robust personality analysis than the fact that the individual is 
willing to share such information in the first place. This insight alerted the team to a new genre of 
powerful behavioral metrics. Instead of analyzing the content of user lists, such as favorite TV shows, 
activities, and music, they learned that simple “meta-data”—such as the amount of information shared 
—‘“turned out to be much more useful and predictive than the original raw data.” The computations 
produced on the strength of these behavioral metrics, when combined with automated linguistic analysis 
and internal Facebook statistics, led the research team to conclude that “we can predict a user’s score for 
a personality trait to within just more than one-tenth of its actual value.”2! The University of Maryland 
team began what would become a multiyear journey toward the instrumentalization of data from the 
depths for the purposes of a highly intentional program of manipulation and behavioral modification. 
Although they could not see very far down that road, they nevertheless anticipated the utility of their 
findings for an eager audience of surveillance capitalists: 


With the ability to infer a user’s personality, social media websites, e-commerce retailers, and even ad servers can be tailored to 
reflect the user’s personality traits and present information such that users will be most receptive to it.... The presentation of 


Facebook ads could be adjusted based on the personality of the user.... Product reviews from authors with personality traits similar to 


the user could be highlighted to increase trust and perceived usefulness.... 52 


The new capabilities also proved robust when applied to other sources of social media meta-data. 
Later that year, the Maryland team published findings that used publicly available Twitter data to predict 
scores on each of the five personality dimensions to within 11-18 percent of their actual value. Similar 
research findings would become central to the progress of rendering Facebook profiles as behavior for 
new caches of surplus from the depths.°3 

In the UK a team of researchers, including Cambridge University’s Michal Kosinski and the deputy 
director of Cambridge’s Psychometrics Centre, David Stillwell, built on this line of research. Stillwell 
had already developed the myPersonality database, a “third-party’ Facebook application that allows 


users to take psychometric tests, like those based on the five-factor model, and receive feedback on their 
results. Launched in 2007 and hosted at the Psychometrics Centre, by 2016 the database contained more 
than six million personality profiles complemented by four million individual Facebook profiles. Once 
regarded as a unique, if offbeat, source of psychological data, myPersonality had become the database of 
choice for the scoping, standardization, and validation of the new models capable of predicting 
personality values from ever-smaller samples of Facebook data and meta-data. Later, it would become 
the model for the work of a small consultancy called Cambridge Analytica, which used the new caches of 
behavioral surplus for an onslaught of politically inspired behavioral micro-targeting. 

In a 2012 paper Kosinski and Stillwell concluded that “user personality can be easily and effectively 
predicted from public data” and warned that social media users are dangerously unaware of the 
vulnerabilities that follow their innocent but voluminous personal disclosures. Their discussion 
specifically cited Facebook CEO Mark Zuckerberg’s unilateral upending of established privacy norms in 
2010, when he famously announced that Facebook users no longer have an expectation of privacy. 
Zuckerberg had described the corporation’s decision to unilaterally release users’ personal information, 
declaring, “We decided that these would be the social norms now, and we just went for it.”2 

Despite their misgivings, the authors went on to suggest the relevance of their findings for “marketing,” 
“user interface design,” and recommender systems.2° In 2013 another provocative study by Kosinski, 
Stillwell, and Microsoft’s Thore Graepel revealed that Facebook “likes” could “automatically and 
accurately estimate a wide range of personal attributes that people would typically assume to be private,” 
including sexual orientation, ethnicity, religious and political views, personality traits, intelligence, 
happiness, use of addictive substances, parental separation, age, and gender. 

The authors appeared increasingly ambivalent about the social implications of their work. On the one 
hand, they announced that these new predictive capabilities could be used to “improve numerous products 
and services.” They concluded that online businesses can adjust their behavior to match each user’s 
personality, with marketing and product recommendations psychologically tailored to each individual. But 
the authors also warned that automated prediction engines run by companies, governments, or Facebook 
itself can compute millions of profiles without individual consent or awareness, discovering facts “that an 
individual may not have intended to share.” The researchers cautioned that “one can imagine situations in 
which such predictions, even if incorrect, could pose a threat to an individual’s well-being, freedom, or 
even life.”28 

Despite these ethical quandaries, by 2015 Kosinski had moved to Stanford University (first to the 
Computer Science Department and then to the Graduate School of Business), where his research quickly 
attracted funding from the likes of Microsoft, Boeing, Google, the National Science Foundation, and the 
Defense Advanced Research Projects Agency (DARPA).°2 Kosinksi and a variety of collaborators, often 
including Stillwell, went on to publish a succession of articles that elaborated and extended the 
capabilities demonstrated in the early papers, refining procedures that “quickly and cheaply assess large 
groups of participants with minimal burden.”© 

A paper published in 2015 broke fresh ground again by announcing that the accuracy of the team’s 
computer predictions had equaled or outpaced that of human judges, both in the use of Facebook “likes” 
to assess personality traits based on the five-factor model and to predict “life outcomes” such as “life 
satisfaction,” “substance use,” or “depression.”®! The study made clear that the real breakthrough of the 
Facebook prediction research was the achievement of economies in the exploitation of these most- 
intimate behavioral depths with “automated, accurate, and cheap personality assessment tools” that 
effectively target a new class of “objects” once known as your “personality.”®2 That these economies can 
be achieved outside the awareness of unrestrained animals makes them even more appealing; as one 


research team emphasizes, “The traditional method for personality evaluation is extremely costly in terms 
of time and labor, and it cannot acquire customer personality information without their awareness....” 

Personality analysis for commercial advantage is built on behavioral surplus—the so-called meta-data 
or mid-level metrics—honed and tested by researchers and destined to foil anyone who thinks that she is 
in control of the “amount” of personal information that she reveals in social media. In the name of, for 
example, affordable car insurance, we must be coded as conscientious, agreeable, and open. This is not 
easily faked because the surplus retrieved for analysis is necessarily opaque to us. We are not scrutinized 
for substance but for form. The price you are offered does not derive from what you write about but how 
you write it. It is not what is in your sentences but in their length and complexity, not what you list but that 
you list, not the picture but the choice of filter and degree of saturation, not what you disclose but how you 
share or fail to, not where you make plans to see your friends but how you do so: a casual “later” or a 
precise time and place? Exclamation marks and adverb choices operate as revelatory and potentially 
damaging signals of your self. 

That the “personality” insights themselves are banal should not distract us from the fact that the volume 
and depth of the new surplus supplies enabled by these extraction operations are unprecedented; nothing 
like this has ever been conceivable.“ As Kosinski told an interviewer in 2015, few people understand 
that companies such as “Facebook, Snapchat, Microsoft, Google and others have access to data that 
scientists would not ever be able to collect.”® Data scientists have successfully predicted traits on the 
five-factor personality model with surplus culled from Twitter profile pictures (color, composition, image 
type, demographic information, facial presentation, and expression...), selfies (color, photographic styles, 
visual texture...), and Instagram photos (hue, brightness, saturation...). Others have tested alternative 
algorithmic models and personality constructs. Another research team demonstrated the ability to predict 
“satisfaction with life” from Facebook messages. This new world has no manager on his or her hands 
and knees scissoring pages of computer conferencing messages into thematically organized piles of paper 
scraps. It’s not the office floor that is crawled. It is you. 

In his 2015 interview, Kosinski observed that “all of our interactions are being mediated through 
digital products and services which basically means that everything is being recorded.” He even 
characterized his own work as “pretty creepy”: “I actually want to stress that I think that many of the 
things that... one can do should certainly not be done by corporations or governments without users’ 
consent.” Recognizing the woefully asymmetric division of learning, he lamented the refusals of Facebook 
and the other internet firms to share their data with the “general public,” concluding that “it’s not because 
they’re evil, but because the general public is bloody stupid... as a society we lost the ability to convince 
large companies that have enormous budgets and enormous access to data to share this goodness with us. 
... We should basically grow up finally and stop it.”© 

In capitalism, though, latent demand summons suppliers and supplies. Surveillance capitalism is no 
different. The prediction imperative unleashes the surveillance hounds to stalk behavior from the depths, 
and well-intentioned researchers unwittingly oblige, leaving a trail of cheap, push-button raw meat for 
surveillance capitalists to hunt and devour. It did not take long. By early 2015, IBM announced that its 
Watson Personality Service was open for business.®8 The corporation’s machine intelligence tools are 
even more complex and invasive than those used in most academic studies. In addition to the five-factor 
personality model, IBM assesses each individual across twelve categories of “needs,” including 
“Excitement, Harmony, Curiosity, Ideal, Closeness, Self-expression, Liberty, Love, Practicality, Stability, 
Challenge, and Structure.” It then identifies “values,” defined as “motivating factors which influence a 
person’s decision-making across’ five dimensions: Self-transcendence/Helping others, 
Conservation/Tradition, Hedonism/Taking pleasure in life, Self-enhancement/Achieving success, and 


Open to change/Excitement.”& 


IBM promises “limitless” applications of its new surplus supplies and “deeper portraits of individual 
customers.” As we would expect, these operations are tested among captive employees who, once 
habituated, can become docile members of a behaviorally purified society. “Personality correlates” can 
now be identified that predict the precise ways in which each customer will react to marketing efforts. 
Who will redeem a coupon? Who will purchase which product? The corporation says that “social media 
content and behavior” can be used to “capitalize on targeted revenue-generating opportunities” with 
“mapping rules from personality to behavior.” The messaging and approach of customer service agents, 
insurance agents, travel agents, real estate agents, investment brokers, and so on can be “matched” to the 
“personality” of the customer, with those psychological data displayed to the agent at the precise moment 
of contact? IBM’s research demonstrates that agents who express personality traits associated with 
“agreeableness” and “conscientiousness” produce significantly higher levels of customer satisfaction. It is 
common sense, except that now these interactions are measured and monitored in real time and at scale, 
with a view to rewarding or extinguishing behavior according to its market effect. 

Thanks to rendition, a handful of now-measurable personal characteristics, including the “need for 
love,” predict the likelihood of “liking a brand.” In a Twitter targeted-ad experiment, IBM found that it 
could significantly increase click-through rates and “follow” rates by targeting individuals with high 
“openness” and low “neuroticism” scores on the five-factor personality analysis. In another study, IBM 
rendered behavioral data from 2,000 Twitter users to establish metrics such as response rates, activity 
levels, and elapsed time between tweets, in addition to psycholinguistic analyses of tweets and five- 
factor personality analysis. IBM “trained” its predictive model by asking the 2,000 users either location- 
related or product-related questions. The findings showed that personality information predicted the 
likelihood of responses. People whom the machines rated as moral, trusting, friendly, extroverted, and 
agreeable tended to respond, compared to low response rates from people rated as cautious and anxious. 
Many of the characteristics that we try to teach our children and model in our own behavior are simply 
repurposed as dispossession opportunities for hidden machine processes of rendition. In this new world, 
paranoia and anxiety function as sources of protection from machine invasion for profit. Must we teach 
our children to be anxious and suspicious? 

IBM is not alone, of course. An innovative breed of personality mercenaries quickly set to work 
institutionalizing the new supply operations. Their efforts suggest how quickly we lose our bearings as 
institutionalization first establishes a sense of normalcy and social acceptance and then gradually 
produces the numbness that accompanies habituation. This process begins with business plans and 
marketing messages, new products and services, and journalistic representations that appear to accept the 
new facts as given.2 

Among this new cohort of mercenaries was Cambridge Analytica, the UK consulting firm owned by 
the reclusive billionaire and Donald Trump backer Robert Mercer. The firm’s CEO, Alexander Nix, 
boasted of its application of personality-based “micro-behavioral targeting” in support of the “Leave” 
and the Trump campaigns during the ramp-up to the 2016 Brexit vote and the US presidential election.4 
Nix claimed to have data resolved “to an individual level where we have somewhere close to four or five 
thousand data points on every adult in the United States.” While scholars and journalists tried to 
determine the truth of these assertions and the role that these techniques might have played in both 2016 
election upsets, the firm’s new chief revenue officer quietly announced the firm’s less glamorous but more 
lucrative postelection strategy: “After this election, it'll be full-tilt into the commercial business.” Writing 
in a magazine for car dealers just after the US election, he tells them that his new analytic methods reveal 
“how a customer wants to be sold to, what their personality type is, and which methods of persuasion are 


most effective.... What it does is change people’s behavior through carefully crafted messaging that 
resonates with them.... It only takes small improvements in conversion rates for a dealership to see a 
dramatic shift in revenue.” 

A leaked Facebook document acquired in 2018 by the Intercept illustrates the significance of data 
drawn from the depths in the fabrication of Facebook’s prediction products, confirms the company’s 
primary orientation to its behavioral futures markets, and reveals the degree to which Cambridge 
Analytica’s controversial practices reflected standard operating procedures at Facebook.“ The 
confidential document cites Facebook’s unparalleled “machine learning expertise” aimed at meeting its 
customers’ “core business challenges.” To this end it describes Facebook’s ability to use its unrivaled 
and highly intimate data stores “to predict future behavior,” targeting individuals on the basis of how they 
will behave, purchase, and think: now, soon, and later. The document links prediction, intervention, and 
modification. For example, a Facebook service called “loyalty prediction” is touted for its ability to 
analyze behavioral surplus in order to predict individuals who are “at risk” of shifting their brand 
allegiance. The idea is that these predictions can trigger advertisers to intervene promptly, targeting 
aggressive messages to stabilize loyalty and thus achieve guaranteed outcomes by altering the course of 
the future. 

Facebook’s “prediction engine” is built on a new machine intelligence platform called “FBLearner 
Flow,” which the company describes as its new “AI backbone” and the key to “personalized experiences” 
that deliver “the most relevant content.” The machine learning system “ingests trillions of data points 
every day, trains thousands of models—either offline or in real time—and then deploys them to the server 
fleet for live predictions.” The company explains that “since its inception, more than a million models 
have been trained, and our prediction service has grown to make more than 6 million predictions per 
second.”2 

As we have already seen, “personalization” derives from prediction, and prediction derives from ever 
richer sources of behavioral surplus and therefore ever more ruthless rendition operations. Indeed, the 
confidential document cites some of the key raw materials fed into this high-velocity, high-volume, and 
deeply scoped manufacturing operation, including not only location, Wi-Fi network details, and device 
information but also data from videos, analyses of affinities, details of friendships, and similarities with 
friends. 

It was probably no coincidence that the leaked Facebook presentation appeared around the same time 
that a young Cambridge Analytica mastermind-turned-whistleblower, Chris Wylie, unleashed a torrent of 
information on that company’s secret efforts to predict and influence individual voting behavior, quickly 
riveting the world on the small political analytics firm and the giant source of its data: Facebook. There 
are many unanswered questions about the legality of Cambridge Analytica’s complex subterfuge, its actual 
political impact, and its relationship with Facebook. Our interest here is restricted to how its 
machinations shine a bright light on the power of surveillance capitalism’s mechanisms, especially the 
determination to render data from the depth dimension. 

Kosinski and Stillwell had called attention to the commercial value of their methods, understanding 
that surplus from the depths afforded new possibilities for behavioral manipulation and modification. 
Wylie recounts his fascination with this prospect, and, through a complicated chain of events, it was he 
who persuaded Cambridge Analytica to use Kosinski and Stillwell’s data to advance its owner’s political 
aims. The objective was “behavioral micro-targeting... influencing voters based not on their 
demographics but on their personalities....”22 When negotiations with Kosinski and Stillwell broke down, 
a third Cambridge academic, Alexander Kogan, was hired to render a similar cache of Facebook 
personality data. 


Kogan was well-known to Facebook. He had collaborated with its data scientists on a 2013 project in 
which the company provided data on 57 billion “friendships.” This time, he paid approximately 270,000 
people to take a personality quiz. Unknown to these participants, Kogan’s app enabled him to access their 
Facebook profiles and, on average, the profiles of about 160 of each of the test takers’ friends, “none of 
whom would have known or had reason to suspect” this invasion.£2 It was a massive rendition operation 
from which Kogan successfully produced psychological profiles of somewhere between 50 and 87 
million Facebook users, data that he then sold to Cambridge Analytica.2! When Facebook questioned him 
about his application, Kogan vowed that his research was solely for academic purposes. Indeed, mutual 
respect between the two parties was sufficiently robust that Facebook hired one of Kogan’s assistants to 
join its in-house team of research psychologists. 

“We exploited Facebook to harvest millions of people’s profiles,” Wylie admitted, “and built models 
to exploit what we knew about them and target their inner demons.” His summary of Cambridge 
Analytica’s accomplishments is a précis of the surveillance capitalist project and a rationale for its 
determination to render from the depths. These are the very capabilities that have gathered force over the 
nearly two decades of surveillance capitalism’s incubation in lawless space. These practices produced 
outrage around the world, when in fact they are routine elements in the daily elaboration of surveillance 
capitalism’s methods and goals, both at Facebook and within other surveillance capitalist companies. 
Cambridge Analytica merely reoriented the surveillance capitalist machinery from commercial markets in 
behavioral futures toward guaranteed outcomes in the political sphere. It was Eric Schmidt, not Wylie, 
who first pried open this Pandora’s box, paving the way for the transfer of surveillance capitalism’s core 
mechanisms to the electoral process as he cemented the mutual affinities that produced surveillance 
exceptionalism. In fact, Wylie enjoyed his early training under Obama’s “director of targeting.” 
Schmidt’s now-weaponized innovations have become the envy of every political campaign and, more 
dangerously, of every enemy of democracy. 

In addition to employing surveillance capitalism’s foundational mechanisms—rendition, behavioral 
surplus, machine intelligence, prediction products, economies of scale, scope, and action—Cambridge 
Analytica’s dark adventure also exemplifies surveillance capitalism’s tactical requirements. Its 
operations were designed to produce ignorance through secrecy and the careful evasion of individual 
awareness. Wylie calls this “information warfare,” correctly acknowledging the asymmetries of 
knowledge and power that are essential to the means of behavioral modification: 


I think it’s worse than bullying, because people don’t necessarily know it’s being done to them. At least bullying respects the agency 
of people because they know... if you do not respect the agency of people, anything that you’re doing after that point is not 


conducive to a democracy. And fundamentally, information warfare is not conducive to democracy.82 


This “warfare” and its structure of invasion and conquest represent surveillance capitalism’s standard 
operating procedures to which billions of innocents are subjected each day, as rendition operations 
violate all boundaries and modification operations claim dominion over all people. Surveillance 
capitalism imposes this quid pro quo of “agency” as the price of information and connection, continuously 
pushing the envelope of rendition to new frontiers. Along the way, companies such as Facebook and 
Google employ every useful foot soldier, including social scientists such as Kogan who are willing to put 
their shoulders to the wheel as they help the company learn, perfect, and integrate the cutting-edge 
methods that can conquer the next frontier, a phenomenon that we will visit in more depth in Chapter 10. 

Irrespective of Cambridge Analytica’s actual competence and its ultimate political impact, the plotting 
and planning behind its ambitions are testament to the pivotal role of rendition from the depths in the 


prediction and modification of behavior, always in pursuit of certainty. Billionaires such as Zuckerberg 
and Mercer have discovered that they can muscle their way to dominance of the division of learning in 
society by setting their sights on these rendition operations and the fortunes they tell. They aim to be 
unchallenged in their power to know, to decide who knows, and to decide who decides. The rendition of 
“personality” was an important milestone in this quest: a frontier, yes, but not the final frontier. 


III. Machine Emotion 


In 2015 an eight-year-old startup named Realeyes won a 3.6 million euro grant from the European 
Commission for a project code-named “SEWA: Automatic Sentiment Analysis in the Wild.” The aim was 
“to develop automated technology that will be able to read a person’s emotion when they view content 
and then establish how this relates to how much they liked the content.” The director of video at AOL 
International called the project “a huge leap forward in video ad tech” and “the Holy Grail of video 
marketing.”8° Just a year later, Realeyes won the commission’s Horizon 2020 innovation prize thanks to 
its “machine learning-based tools that help market researchers analyze the impact of their advertising and 
make it more relevant.” 

The SEWA project is a window on a burgeoning new domain of rendition and behavioral surplus 
supply operations known as “affective computing,” “emotion analytics,” and “sentiment analysis.” The 
personalization project descends deeper toward the ocean floor with these new tools, where they lay 
claim to yet a new frontier of rendition trained not only on your personality but also on your emotional 
life. If this project of surplus from the depths is to succeed, then your unconscious—where feelings form 
before there are words to express them—must be recast as simply one more source of raw-material 
supply for machine rendition and analysis, all of it for the sake of more-perfect prediction. As a market 
research report on affective computing explains, “Knowing the real-time emotional state can help 
businesses to sell their product and thereby increase revenue.”®2 

Emotion analytics products such as SEWA use specialized software to scour faces, voices, gestures, 
bodies, and brains, all of it captured by “biometric” and “depth” sensors, often in combination with 
imperceptibly small, “unobtrusive” cameras. This complex of machine intelligence is trained to isolate, 
capture, and render the most subtle and intimate behaviors, from an inadvertent blink to a jaw that 
slackens in surprise for a fraction of a second. Combinations of sensors and software can recognize and 
identify faces; estimate age, ethnicity, and gender; analyze gaze direction and blinks; and track distinct 
facial points to interpret “micro-expressions,” eye movements, emotions, moods, stress, deceit, boredom, 
confusion, intentions, and more: all at the speed of life.22 As the SEWA project description says, 


Technologies that can robustly and accurately analyse human facial, vocal and verbal behaviour and interactions in the wild, as 
observed by omnipresent webcams in digital devices, would have profound impact on both basic sciences and the industrial sector. 


They... measure behaviour indicators that heretofore resisted measurement because they were too subtle or fleeting to be measured 


by the human eye and eat... 20 


These behaviors also elude the conscious mind. The machines capture the nanosecond of disgust that 
precedes a rapid-fire sequence of anger, comprehension, and finally joy on the face of a young woman 
watching a few frames of film, when all she can think to say is “I liked it!” A Realeyes white paper 
explains that its webcams record people watching videos in their homes “so we can capture genuine 
reactions.” Algorithms process facial expressions, and “emotions are detected, aggregated, and reported 


online in real time, second by second... enabling our clients to make better business decisions.” Realeyes 
emphasizes its own “proprietary metrics” to help marketers “target audiences” and “predict 
performance.” 

Once again, a key theme of machine intelligence is that quality is a function of quantity. Realeyes says 
that its data sets contain over 5.5 million individually annotated frames of more than 7,000 subjects from 
all over the world: “We are continuously working to build the world’s largest expression and behaviour 
datasets by increasing the quality and volume of our already-existing categories, and by creating new sets 
—for other expressions, emotions, different behavioral clues or different intensities.... Having automated 
this process, it can then be scaled up to simultaneously track the emotions of entire audiences.”2 Clients 
are advised to “play your audience emotions to stay on top of the game.”22 The company’s website offers 
a brief review of the history of research on human emotions, concluding that “the more people feel, the 
more they spend.... Intangible ‘emotions’ translate into concrete social activity, brand awareness, and 
profit.” 

The chair of SEWA’s Industry Advisory Board is frank about this undertaking, observing that 
unlocking the meaning of “the non-spoken language of the whole body and interpreting complex emotional 
response... will be wonderful for interpreting reactions to marketing materials,” adding that “it is simply 
foolish not to take emotional response into account when evaluating all marketing materials.” Indeed, 
these “nonconscious tools” extract rarified new qualities of behavioral surplus from your inner life in 
order to predict what you will buy and the precise moment at which you are most vulnerable to a push. 
SEWA’s advisory chair says that emotional analytics are “like identifying individual musical notes.” Each 
potential customer, then, is a brief and knowable composition: “We will be able to identify chords of 
human response such as ‘liking,’ boredom, etc.... We will ultimately become masters of reading each 
other’s feelings and intent.”2° 

This is not the first time that the unconscious mind has been targeted as an instrument of others’ aims. 
Propaganda and advertising have always been designed to appeal to unacknowledged fears and yearnings. 
These have relied more on art than science, using gross data or professional intuition for the purpose of 
mass communication.2° Those operations cannot be compared to the scientific application of today’s 
historic computational power to the micro-measured, continuous rendition of your more-or-less actual 
feelings. The new toolmakers do not intend to rob you of your inner life, only to surveil and exploit it. All 
they ask is to know more about you than you know about yourself. 

Although the treasures of the unconscious mind have been construed differently across the millennia— 
from spirit to soul to self—the ancient priest and the modern psychotherapist are united in an age-old 
reverence for its primal healing power through self-discovery, self-mastery, integration, restoration, and 
transcendence. In contrast, the conception of emotions as observable behavioral data first took root in the 
mid-1960s with the work of Paul Ekman, then a young professor at the University of California, San 
Francisco. From his earliest papers, Ekman argued that “actions speak louder than words.”2’ Even when a 
person is determined to censor or control his or her emotional communications, Ekman postulated that 
some types of nonverbal behaviors “escape control and provide leakage.”2° Early on, he recognized the 
potential utility of a “categorical scheme” that reliably traced the effects of expression back to their 
causes in emotion, and in 1978 Ekman, along with frequent collaborator Wallace Friesen, published the 
seminal Facial Action Coding System (FACS), which provided that scheme. 

FACS distinguishes the elemental movements of facial muscles, breaking them down into twenty-seven 
facial “action units,” along with more for the head, eyes, tongue, and so on. Later, Ekman concluded that 
six “basic emotions” (anger, fear, sadness, enjoyment, disgust, and surprise) anchored the wider array of 
human emotional expression.122 FACS and the six-emotion model became the dominant paradigm for the 


study of facial expression and emotion, in much the same way that the five-factor model came to dominate 
studies of personality. 

The program of emotional rendition began innocently enough with MIT Media Lab professor Rosalind 
Picard and the new field of computer science that she called “affective computing.” She was among the 
first to recognize the opportunity for a computational system to automate the analysis of Ekman’s facial 
configurations and correlate micro-expressions with their emotional causality12! She aimed to combine 
facial expression with the computation of vocal intonation and other physiological signals of emotion that 
could be measured as behavior. In 1997 she published Affective Computing, which posited a practical 
solution to the idea that some emotions are available to the conscious mind and can be expressed 
“cognitively” (“I feel scared”), whereas others may elude consciousness but nevertheless be expressed 
physically in beads of sweat, a widening of the eyes, or a nearly imperceptible tightening of the jaw. 

The key to affective computing, Picard argued, was to render both conscious and unconscious emotion 
as observable behavior for coding and calculation. A computer, she reasoned, would be able to render 
your emotions as behavioral information. Affect recognition, as she put it, is “a pattern recognition 
problem,” and “affect expression” is pattern synthesis. The proposition was that “computers can be given 
the ability to recognize emotions as well as a third-person human observer.” 

Picard imagined her affective insights being put to use in ways that were often good-hearted or, at the 
very least, benign. Most of the applications she describes conform to the logic of the Aware Home: any 
knowledge produced would belong to the subject to enhance his or her reflexive learning. For example, 
she envisioned a “computer-interviewing agent” that could function as an “affective mirror” coaching a 
student in preparation for a job interview or a date and an automatic agent that could alert you to hostile 
tones in your own prose before you press “send.” Picard anticipated other tools combining software and 
sensors that she believed could enhance daily life in a range of situations, such as helping autistic children 
develop emotional skills, providing software designers with feedback on users’ frustration levels, 
assigning points to video game players to reward courage or stress reduction, producing learning modules 
that stimulate curiosity and minimize anxiety, and analyzing emotional dynamics in a classroom. She 
imagined software agents that learn your preferences and find you the kinds of news stories, clothing, art, 
or music that make you smile.122 Whatever one’s reaction to these ideas might be, they share one key 
pattern: unlike SEWA’s model, Picard’s data were intended to be for you, not merely about you. 

Back in 1997, Picard acknowledged the need for privacy “so that you remain in control over who gets 
access to this information.” Importantly for our analysis, in the final pages of her book she expressed 
some concerns, writing that “there are good reasons not to broadcast your affective patterns to the 
world... you might flaunt your good mood in front of friends... you probably do not want it picked up by 
an army of sales people who are eager to exploit mood-based buying habits, or by advertisers eager to 
convince you that you’d feel better if you tried their new soft drink right now.” She noted the possibility of 
intrusive workplace monitoring, and she voiced reservations about the possibility of a dystopian future in 
which “malevolent” governmental forces use affective computing to manipulate and control the emotions 
of populations.122 

Despite these few paragraphs of apprehension, her conclusions were bland. Every technology arrives 
with its “pros and cons,” she wrote. The concerns are not “insurmountable” because “safeguards can be 
developed.” Picard was confident that technologies and techniques could solve any problem, and she 
imagined “wearable computers” that “gather information strictly for your own use....” She stressed the 
importance of ensuring “that the wearer retains ultimate control over the devices he chooses to wear, so 
that they are tools of helpful empowerment and not of harmful subjugation.”2%4 

In a pattern that is by now all too familiar, safeguards lagged as surveillance capitalism flourished. By 


early 2014, Facebook had already applied for an “emotion detection” patent designed to implement each 
of Picard’s fears122 The idea was “one or more software modules capable of detecting emotions, 
expressions or other characteristic of a user from image information.” As always, the company was 
ambitious. Its list of detectable emotions “by way of example and not limitation” included expressions 
such as “a smile, joy, humor, amazement, excitement, surprise, a frown, sadness, disappointment, 
confusion, jealousy, indifference, boredom, anger, depression, or pain.” The hope was that “over time” 
their module would be able to assess “a user’s interest in displayed content” for the purposes of 
“customization based on emotion type.”222 

By 2017, exactly twenty years after the publication of Picard’s book, a leading market research firm 
forecast that the “affective computing market,” including software that recognizes speech, gesture, and 
facial expressions along with sensors, cameras, storage devices, and processors, would grow from $9.35 
billion in 2015 to $53.98 billion in 2021, predicting a compounded annual growth rate of nearly 35 
percent. What happened to cause this explosion? The report concludes that heading up the list of 
“triggers” for this dramatic growth is “rising demand for mapping human emotions especially by the 
marketing and advertising sector....”424 Picard’s good intentions were like so many innocent iron filings 
in the presence of a magnet as the market demand exerted by the prediction imperative drew affective 
computing into surveillance capitalism’s powerful force field. 

Picard would eventually become part of this new dispossession industry with a company called 
Affectiva that was cofounded with Rana el Kaliouby, an MIT Media Lab postdoctoral research scientist 
and Picard protégé. The company’s transformation from doing good to doing surveillance capitalism is a 
metaphor for the fate of the larger undertaking of emotion analysis as it is rapidly drawn into the 
competitive maelstrom for surveillance revenues. 

Both Picard and Kaliouby had shared a vision of applying their research in medical and therapeutic 
settings. The challenges of autistic children seemed a perfect fit for their discoveries, so they trained a 
machine system called MindReader to recognize emotions using paid actors to mimic specific emotional 
responses and facial gestures. Early on, MIT Media Lab corporate sponsors Pepsi, Microsoft, Bank of 
America, Nokia, Toyota, Procter and Gamble, Gillette, Unilever, and others had overwhelmed the pair 
with queries about using their system to measure customers’ emotional responses. Kaliouby describes the 
women’s hesitation and their determination to focus on “do-good” applications. According to her account, 
the Media Lab encouraged the two women to “spin off” their work into the startup they called Affectiva, 
imagined as a “baby IBM for emotionally intelligent machines.”222 

It wasn’t long before the new company found itself fielding significant interest from ad agencies and 
marketing firms itching for automated rendition and analysis from the depths. Describing that time, as 
Picard told one journalist, “Our CEO was absolutely not comfortable with the medical space.” As a 
result, Picard was “pushed out” of the firm three years after its founding. As an Affectiva researcher 
recounted, “We began with a powerful set of products that could assist people who have a very difficult 
time with perceiving affect.... Then they started to emphasize only the face, to focus on advertisements, 
and on predicting whether someone likes a product, and just went totally off the original mission.” 22 

Companies such as market research firm Millward Brown and advertising powerhouse McCann 
Erickson, competing in a new world of targeted “personalized” ads, already craved access to the 
inarticulate depths of the consumer response. Millward Brown had even formed a neuroscience unit but 
found it impossible to scale. It was Affectiva’s analysis of one particularly nuanced ad for Millward 
Brown that dazzled its executives and decisively turned the tide for the startup. “The software was telling 
us something we were potentially not seeing,” one Millward Brown executive said. “People often can’t 
articulate such detail in sixty seconds.” 2 


By 2016, Kaliouby was the company’s CEO, redefining its business as “Emotion AI” and calling it 
“the next frontier of artificial intelligence.”! The company had raised $34 million in venture capital, 
included 32 Fortune 100 companies and 1,400 brands from all over the world among its clients, and 
claimed to have the largest repository of emotion data in the world, with 4.8 million face videos from 75 
countries, even as it continued to expand its supply routes with data sourced from online viewing, video 
game participation, driving, and conversation.4 

This is the commercial context in which Kaliouby came to feel that it is perfectly reasonable to assert 
that an “emotion chip” will become the base operational unit of a new “emotion economy.” She speaks to 
her audiences of a chip embedded in all things everywhere, running constantly in the background, 
producing an “emotion pulse” each time you check your phone: “I think in the future we’ll assume that 
every device just knows how to read your emotions.”/? At least one company, Emoshape, has taken her 
proposition seriously. The firm, whose tagline is “Life Is the Value,” produces a microchip that it calls 
“the industry’s first emotion synthesis engine,” delivering “high performance machine emotion 
awareness.” The company writes that its chip can classify twelve emotions with up to 98 percent 
accuracy, enabling its “artificial intelligence or robot to experience 64 trillion possible distinct emotional 
states,” 114 

Kaliouby imagines that pervasive “emotion scanning” will come to be as taken for granted as a 
“cookie” planted in your computer to track your online browsing, After all, those cookies once stirred 
outrage, and now they inundate every online move. For example, she anticipates YouTube scanning its 
viewers’ emotions as they watch videos. Her confidence is buoyed by demand that originates in the 
prediction imperative: “The way I see it, it doesn’t matter that your Fitbit doesn’t have a camera, because 
your phone does, and your laptop does, and your TV will. All that data gets fused with biometrics from 
your wearable devices and builds an emotional profile for you.” As a start, Affectiva pioneered the 
notion of “emotion as a service,” offering its analytics on demand: “Just record people expressing 
emotion and then send those videos or images to us to get powerful emotion metrics back.” 

The possibilities in the depth dimension seem endless, and perhaps they will be if Affectiva, its 
clients, and fellow travelers are free to plunder our selves at will. There are indications of more far- 
reaching ambitions in which “emotion as a service” expands from observation to modification. 
“Happiness as a service” seems to be within reach. “I do believe that if we have information about your 
emotional experiences we can help you be in a positive mood,” Kaliouby says. She imagines emotion- 
recognition systems issuing reward points for happiness because, after all, happy customers are more 
“engaged.” Ho 


IV. When They Come for My Truth 


Rendition is by now a global project of surveillance capital, and in the depth dimension we see it at its 
most pernicious. Intimate territories of the self, like personality and emotion, are claimed as observable 
behavior and coveted for their rich deposits of predictive surplus. Now the personal boundaries that 
shelter inner life are officially designated as bad for business by a new breed of mercenaries of the self 
determined to parse and package inner life for the sake of surveillance revenues. Their expertise disrupts 
the very notion of the autonomous individual by rewarding “boundarylessness” with whatever means are 
available—offers of elite status, bonuses, happiness points, discounts, “buy” buttons pushed to your 
device at the precise moment predicted for maximum success—so that we might strip and surrender to the 
pawing and prying of the machines that serve the new market cosmos. 


I want to deliberately sidestep a more detailed discussion of what is “personality” or “emotion,” 
“conscious” or “unconscious,” in favor of what I hope is a less fractious truth thrown into relief by this 
latest phase of incursion. Experience is not what is given to me but rather what I make of it. The same 
experience that I deride may invite your enthusiasm. The self is the inward space of lived experience from 
which such meanings are created. In that creation I stand on the foundation of personal freedom: the 
“foundation” because I cannot live without making sense of my experience. 

No matter how much is taken from me, this inward freedom to create meaning remains my ultimate 
sanctuary. Jean-Paul Sartre writes that “freedom is nothing but the existence of our will,” and he 
elaborates: “Actually it is not enough to will; it is necessary to will to will.”4 This rising up of the will 
to will is the inner act that secures us as autonomous beings who project choice into the world and 
exercise the qualities of self-determining moral judgment that are civilization’s necessary and final 
bulwark. This is the sense behind another of Sartre’s insights: “Without bearings, stirred by a nameless 
anguish, the words labor.... The voice is born of a risk: either to lose oneself or win the right to speak in 
the first person.” 2 

As the prediction imperative drives deeper into the self, the value of its surplus becomes irresistible, 
and cornering operations escalate. What happens to the right to speak in the first person from and as my 
self when the swelling frenzy of institutionalization set into motion by the prediction imperative is trained 
on cornering my sighs, blinks, and utterances on the way to my very thoughts as a means to others’ ends? It 
is no longer a matter of surveillance capital wringing surplus from what I search, buy, and browse. 
Surveillance capital wants more than my body’s coordinates in time and space. Now it violates the inner 
sanctum as machines and their algorithms decide the meaning of my breath and my eyes, my jaw muscles, 
the hitch in my voice, and the exclamation points that I offered in innocence and hope. 

What happens to my will to will myself into the first person when the surrounding market cosmos 
disguises itself as my mirror, shape-shifting according to what it has decided I feel or felt or will feel: 
ignoring, goading, chiding, cheering, or punishing me? Surveillance capital cannot keep from wanting all 
of me as deep and far as it can go. One firm that specializes in “human analytics” and affective computing 
has this headline for its marketing customers: “Get Closer to the Truth. Understand the ‘Why.’” What 
happens when they come for my “truth” uninvited and determined to march through my self, taking the bits 
and pieces that can nourish their machines to reach their objectives? Cornered in my self, there is no 
escape.9 

It appears that questions like these may have come to trouble Picard. In a 2016 lecture she gave in 
Germany titled “Towards Machines That Deny Their Maker,” the bland assertions of her 1997 book that 
“safeguards can be developed,” that additional technologies and techniques could solve any problem, and 
that “wearable computers” would “gather information strictly for your own use” as “tools of helpful 
empowerment and not of harmful subjugation’? had given way to new reflections. “Some organizations 
want to sense human emotions without people knowing or consenting,” she said, “A few scientists want to 
build computers that are vastly superior to humans, capable of powers beyond reproducing their own 
kind... how might we make sure that new affective technologies make human lives better?”14! 

Picard did not foresee the market forces that would transform the rendition of emotion into for-profit 
surplus: means to others’ ends. That her vision is made manifest in thousands of activities should be a 
triumph, but it is diminished by the fact that so many of those activities are now bound to the commercial 
surveillance project. Each failure to establish bearings contributes to habituation, normalization, and 
ultimately legitimation. Subordinated to the larger aims of surveillance capitalism, the thrust of the 
affective project changed as if distorted in a fun-house mirror. 

This cycle calls to mind the words of another MIT professor, the computer scientist and humanist 


Joseph Weizenbaum, who spoke eloquently and often on the inadvertent collusion of computer scientists 
in the construction of terrifying weapons systems. I believe he would have shaken his spear in the 
direction of today’s sometimes-unwitting and sometimes-intentional mercenaries of the self, and it is 
fitting to conclude here with his voice: 


I don’t quite know whether it is especially computer science or its sub-discipline Artificial Intelligence that has such an enormous 
affection for euphemism. We speak so spectacularly and so readily of computer systems that understand, that see, decide, make 
judgments... without ourselves recognizing our own superficiality and immeasurable naiveté with respect to these concepts. And, in 
the process of so speaking, we anesthetize our ability to... become conscious of its end use.... One can’t escape this state without 


asking, again and again: “What do I actually do? What is the final application and use of the products of my work?” and ultimately, 


“Am I content or ashamed to have contributed to this use??122 


CHAPTER TEN 


MAKE THEM DANCE 


But hear the morning’s injured weeping and know why: 
Ramparts and souls have fallen; the will of the unjust 
Has never lacked an engine; still all princes must 
Employ the fairly-noble unifying lie. 


—W. H. AUDEN 
SONNETS FROM CHINA, XI 


I. Economies of Action 


“The new power is action,” a senior software engineer told me. “The intelligence of the internet of things 
means that sensors can also be actuators.” The director of software engineering for a company that is an 
important player in the “internet of things” added, “It’s no longer simply about ubiquitous computing. Now 
the real aim is ubiquitous intervention, action, and control. The real power is that now you can modify 
real-time actions in the real world. Connected smart sensors can register and analyze any kind of behavior 
and then actually figure out how to change it. Real-time analytics translate into real-time action.” The 
scientists and engineers I interviewed call this new capability “actuation,” and they describe it as the 
critical though largely undiscussed turning point in the evolution of the apparatus of ubiquity. 

This actuation capability defines a new phase of the prediction imperative that emphasizes economies 
of action. This phase represents the completion of the new means of behavior modification, a decisive 
and necessary evolution of the surveillance capitalist “means of production” toward a more complex, 
iterative, and muscular operational system. It is a critical achievement in the race to guaranteed outcomes. 
Under surveillance capitalism the objectives and operations of automated behavioral modification are 
designed and controlled by companies to meet their own revenue and growth objectives. As one senior 
engineer told me, 


Sensors are used to modify people’s behavior just as easily as they modify device behavior. There are many great things we can do 
with the internet of things, like lowering the heat in all the houses on your street so that the transformer is not overloaded, or 
optimizing an entire industrial operation. But at the individual level, it also means the power to take actions that can override what you 


are doing or even put you on a path you did not choose. 


The scientists and engineers whom I interviewed identified three key approaches to economies of 


action, each one aimed at achieving behavior modification. The first two I call “tuning” and “herding.” 
The third is already familiar as what behavioral psychologists refer to as “conditioning.” Strategies that 
produce economies of action vary according to the methods with which these approaches are combined 
and the salience of each. 

“Tuning” occurs in a variety of ways. It may involve subliminal cues designed to subtly shape the flow 
of behavior at the precise time and place for maximally efficient influence. Another kind of tuning 
involves what behavioral economists Richard Thaler and Cass Sunstein call the “nudge,” which they 
define as “any aspect of a choice architecture that alters people’s behavior in a predictable way.”! The 
term choice architecture refers to the ways in which situations are already structured to channel attention 
and shape action. In some cases these architectures are intentionally designed to elicit specific behavior, 
such as a classroom in which all the seats face the teacher or an online business that requires you to click 
through many obscure pages in order to opt out of its tracking cookies. The use of this term is another way 
of saying in behaviorist language that social situations are always already thick with tuning interventions, 
most of which operate outside our awareness. 

Behavioral economists argue a worldview based on the notion that human mentation is frail and 
flawed, leading to irrational choices that fail to adequately consider the wider structure of alternatives. 
Thaler and Sunstein have encouraged governments to actively design nudges that adequately shepherd 
individual choice making toward outcomes that align with their interests, as perceived by experts. One 
classic example favored by Thaler and Sunstein is the cafeteria manager who nudges students to healthier 
food choices by prominently displaying the fruit salad in front of the pudding; another is the automatic 
renewal of health insurance policies as a way of protecting individuals who overlook the need for new 
approvals at the end of each year. 

Surveillance capitalists adapted many of the highly contestable assumptions of behavioral economists 
as one cover story with which to legitimate their practical commitment to a unilateral commercial 
program of behavior modification. The twist here is that nudges are intended to encourage choices that 
accrue to the architect, not to the individual. The result is data scientists trained on economies of action 
who regard it as perfectly normal to master the art and science of the “digital nudge” for the sake of their 
company’s commercial interests. For example, the chief data scientist for a national drugstore chain 
described how his company designs automatic digital nudges that subtly push people toward the specific 
behaviors favored by the company: “You can make people do things with this technology. Even if it’s just 
5% of people, you’ve made 5% of people do an action they otherwise wouldn’t have done, so to some 
extent there is an element of the user’s loss of self-control.” 

“Herding” is a second approach that relies on controlling key elements in a person’s immediate 
context. The uncontract is an example of a herding technique. Shutting down a car’s engine irreversibly 
changes the driver’s immediate context, herding her out the car door. Herding enables remote 
orchestration of the human situation, foreclosing action alternatives and thus moving behavior along a path 
of heightened probability that approximates certainty. “We are learning how to write the music, and then 
we let the music make them dance,” an “internet of things” software developer explains, adding, 


We can engineer the context around a particular behavior and force change that way. Context-aware data allow us to tie together 
your emotions, your cognitive functions, your vital signs, etcetera. We can know if you shouldn’t be driving, and we can just shut your 
car down. We can tell the fridge, “Hey, lock up because he shouldn’t be eating,” or we tell the TV to shut off and make you get some 
sleep, or the chair to start shaking because you shouldn’t be sitting so long, or the faucet to turn on because you need to drink more 


water. 


“Conditioning” is a well-known approach to inducing behavior change, primarily associated with the 
famous Harvard behaviorist B. F. Skinner. He argued that behavior modification should mimic the 
evolutionary process, in which naturally occurring behaviors are “selected” for success by environmental 
conditions. Instead of the earlier, more simplistic model of stimulus/response, associated with 
behaviorists such as Watson and Pavlov, Skinner interpolated a third variable: “reinforcement.” In his 
laboratory work with mice and pigeons, Skinner learned how to observe a range of naturally occurring 
behaviors in the experimental animal and then reinforce the specific action, or “operant,” that he wanted 
the animal to reproduce. Ultimately, he mastered intricate designs or “schedules” of reinforcement that 
could reliably shape precise behavioral routines. 

Skinner called the application of reinforcements to shape specific behaviors “operant conditioning.” 
His larger project was known as “behavior modification” or “behavioral engineering,” in which behavior 
is continuously shaped to amplify some actions at the expense of others. In the end the pigeon learns, for 
example, to peck a button twice in order to receive a pellet of grain. The mouse learns his way through a 
complicated maze and back again. Skinner imagined a pervasive “technology of behavior” that would 
enable the application of such methods across entire human populations. 

As the chief data scientist for a much-admired Silicon Valley education company told me, 
“Conditioning at scale is essential to the new science of massively engineered human behavior.” He 
believes that smartphones, wearable devices, and the larger assembly of always-on networked nodes 
allow his company to modify and manage a substantial swath of its users’ behavior. As digital signals 
monitor and track a person’s daily activities, the company gradually masters the schedule of 
reinforcements—rewards, recognition, or praise that can reliably produce the specific user behaviors that 
the company selects for dominance: 


The goal of everything we do is to change people’s actual behavior at scale. We want to figure out the construction of changing a 
person’s behavior, and then we want to change how lots of people are making their day-to-day decisions. When people use our app, 
we can capture their behaviors and identify good and bad [ones]. Then we develop “treatments” or “data pellets” that select good 


behaviors. We can test how actionable our cues are for them and how profitable certain behaviors are for us. 


Although it is still possible to imagine automated behavioral modification without surveillance 
capitalism, it is not possible to imagine surveillance capitalism without the marriage of behavior 
modification and the technological means to automate its application. This marriage is essential to 
economies of action. For example, one can imagine a fitness tracker, a car, or a refrigerator whose data 
and operational controls are accessible exclusively to their owners for the purposes of helping them to 
exercise more often, drive safely, and eat healthily. But as we have already seen in so many domains, the 
rise of surveillance capitalism has obliterated the idea of the simple feedback loop characteristic of the 
behavioral value reinvestment cycle. In the end, it’s not the devices; it’s Max Weber’s “economic 
orientation,” now determined by surveillance capitalism. 

The allure of surveillance revenues drives the continuous accumulation of more and more predictive 
forms of behavioral surplus. The most predictive source of all is behavior that has already been modified 
to orient toward guaranteed outcomes. The fusion of new digital means of modification and new economic 
aims produces whole new ranges of techniques for creating and cornering these new forms of surplus. A 
study called “Behavior Change Implemented in Electronic Lifestyle Activity Monitors” is illustrative. 
Researchers from the University of Texas and the University of Central Florida studied thirteen such 
applications, concluding that the monitoring devices “contain a wide range of behavior change techniques 
typically used in clinical behavior interventions.” The researchers conclude that behavior-change 


operations are proliferating as a result of their migration to digital devices and internet connectivity. They 
note that the very possibility of a simple loop designed by and for the consumer seems hopelessly elusive, 
observing that behavior-change apps “lend themselves... to various types of surveillance” and that 
“official methods” of securely and simply transmitting data “do not appear to currently exist in these 
apps.” 

Remember that Google economist Hal Varian extolled the “new uses” of big data that proceed from 
ubiquitous computer-mediated transactions. Among these he included the opportunity for “continuous 
experimentation.” Varian noted that Google has its engineering and data science teams consistently 
running thousands of “A/B” experiments that rely on randomization and controls to test user reactions to 
hundreds of variations in page characteristics from layout to buttons to fonts. Varian endorsed and 
celebrated this self-authorizing experimental role, warning that all the data in the world “can only 
measure correlation, not causality.”2 Data tell what happened but not why it happened. In the absence of 
causal knowledge, even the best predictions are only extrapolations from the past. 

The result of this conundrum is that the last crucial element in the construction of high-quality 
prediction products—i.e., those that approximate guaranteed outcomes—depends upon causal knowledge. 
As Varian says, “If you really want to understand causality, you have to run experiments. And if you run 
experiments continuously, you can continuously improve your system.”4 

Because the “system” is intended to produce predictions, “continuously improving the system’ means 
closing the gap between prediction and observation in order to approximate certainty. In an analog world, 
such ambitions would be far too expensive to be practical, but Varian observes that in the realm of the 
internet, “experimentation can be entirely automated.” 

Varian awards surveillance capitalists the privilege of the experimenter’s role, and this is presented as 
another casual fait accompli. In fact, it reflects the final critical step in surveillance capitalists’ radical 
self-dealing of new rights. In this phase of the prediction imperative, surveillance capitalists declare their 
right to modify others’ behavior for profit according to methods that bypass human awareness, individual 
decision rights, and the entire complex of self-regulatory processes that we summarize with terms such as 
autonomy and self-determination. 

What follows now are two distinct narratives of surveillance capitalists as “experimenters” who 
leverage their asymmetries of knowledge to impose their will on the unsuspecting human subjects who are 
their users. The experimental insights accumulated through their one-way mirrors are critical to 
constructing, fine-tuning, and exploring the capabilities of each firm’s for-profit means of behavioral 
modification. In Facebook’s user experiments and in the augmented-reality game Pokémon Go (imagined 
and incubated at Google), we see the commercial means of behavioral modification evolving before our 
eyes. Both combine the components of economies of action and the techniques of tuning, herding, and 
conditioning in startling new ways that expose the Greeks secreted deep in the belly of the Trojan horse: 
the economic orientation obscured behind the veil of the digital. 


II. Facebook Writes the Music 


In 2012 Facebook researchers startled the public with an article provocatively titled “A 61-Million- 
Person Experiment in Social Influence and Political Mobilization,” published in the scientific journal 
Nature.? In this controlled, randomized study conducted during the run-up to the 2010 US Congressional 
midterm elections, the researchers experimentally manipulated the social and informational content of 
voting-related messages in the news feeds of nearly 61 million Facebook users while also establishing a 
control group. 

One group was shown a statement at the top of their news feed encouraging the user to vote. It 
included a link to polling place information, an actionable button reading “I Voted,” a counter indicating 
how many other Facebook users reported voting, and up to six profile pictures of the user’s Facebook 
friends who had already clicked the “I Voted” button. A second group received the same information but 
without the pictures of friends. A third control group did not receive any special message. 

The results showed that users who received the social message were about 2 percent more likely to 
click on the “I Voted” button than did those who received the information alone and 0.26 percent more 
likely to click on polling place information. The Facebook experimenters determined that social 
messaging was an effective means of tuning behavior at scale because it “directly influenced political 
self-expression, information seeking and real-world voting behavior of millions of people,” and they 
concluded that “showing familiar faces to users can dramatically improve the effectiveness of a 
mobilization message.” 

The team calculated that the manipulated social messages sent 60,000 additional voters to the polls in 
the 2010 midterm elections, as well as another 280,000 who cast votes as a result of a “social contagion” 
effect, for a total of 340,000 additional votes. In their concluding remarks, the researchers asserted that 
“we show the importance of social influence for effecting behavior change... the results suggest that 
online messages might influence a variety of offline behaviors, and this has implications for our 
understanding of the role of online social media in society....”° 

The experiment succeeded by producing social cues that “suggested” or “primed” users in ways that 
tuned their real-world behavior toward a specific set of actions determined by the “experimenters.” In 
this process of experimentation, economies of action are discovered, honed, and ultimately 
institutionalized in software programs and their algorithms that function automatically, continuously, 
ubiquitously, and pervasively to achieve economies of action. Facebook’s surplus is aimed at solving one 
problem: how and when to intervene in the state of play that is your daily life in order to modify your 
behavior and thus sharply increase the predictability of your actions now, soon, and later. The challenge 
for surveillance capitalists is to learn how to do this effectively, automatically, and, therefore, 
economically, as a former Facebook product manager writes: 


Experiments are run on every user at some point in their tenure on the site. Whether that is seeing different size ad copy, or different 
marketing messages, or different call-to-action buttons, or having their feeds generated by different ranking algorithms.... The 
fundamental purpose of most people at Facebook working on data is to influence and alter people’s moods and behavior. They are 


doing it all the time to make you like stories more, to click on more ads, to spend more time on the site. This is just how a website 


works, everyone does this and everyone knows that everyone does this. 


The Facebook study’s publication evoked fierce debate as experts and the wider public finally began 


to reckon with Facebook’s—and the other internet companies’—unprecedented power to persuade, 
influence, and ultimately manufacture behavior. Harvard’s Jonathan Zittrain, a specialist in internet law, 
acknowledged that it was now possible to imagine Facebook quietly engineering an election, using means 
that its users could neither detect nor control. He described the Facebook experiment as a challenge to 
“collective rights” that could undermine “the right of people as a whole... to enjoy the benefits of a 
democratic process....”2 

Public concern failed to destabilize Facebook’s self-authorizing practice of behavior modification at 
scale. Even as the social influence experiment was being debated in 2012, a Facebook data scientist was 
already collaborating with academic researchers on a new study, “Experimental Evidence of Massive- 
Scale Emotional Contagion Through Social Networks,” submitted to the prestigious Proceedings of the 
National Academy of Sciences in 2013, where it was edited by a well-known Princeton social 
psychologist, Susan Fiske, and published in June 2014. 

This time the experimenters “manipulated the extent to which people (N = 689,003) were exposed to 
emotional expressions in their News Feed.”2 The experiment was structured like one of those allegedly 
benign A/B tests. In this case one group was exposed to mostly positive messages in their news feed and 
the other to predominantly negative messages. The idea was to test whether even subliminal exposure to 
specific emotional content would cause people to change their own posting behavior to reflect that 
content. It did. Whether or not users felt happier or sadder, the tone of their expression changed to reflect 
their news feed. 

The experimental results left no doubt that once again Facebook’s carefully designed, undetectable, 
and uncontestable subliminal cues reached beyond the screen into the daily lives of hundreds of thousands 
of naive users, predictably actuating specific qualities of emotional expression through processes that 
operate outside the awareness of their human targets, just as Stuart MacKay had originally prescribed for 
Galapagos turtles and Canadian elk (see Chapter 7). “Emotional states can be transferred to others via 
emotional contagion, leading people to experience the same emotions without their awareness,” the 
researchers proclaimed. “Online messages influence our experience of emotions, which may affect a 
variety of offline behaviors.” The team celebrated its work as “some of the first experimental evidence to 
support the controversial claims that emotions can spread throughout a network,” and they reflected on the 
fact that even their relatively minimal manipulation had a measurable effect, albeit a small one.12 

What Facebook researchers failed to acknowledge in either experiment is that a person’s 
susceptibility to subliminal cues and his or her vulnerability to a “contagion” effect is largely dependent 
upon empathy: the ability to understand and share in the mental and emotional state of another person, 
including feeling another’s feelings and being able to take another’s point of view—sometimes 
characterized as “affective” or “cognitive” empathy. Psychologists have found that the more a person can 
project himself or herself into the feelings of another and take the other’s perspective, the more likely he 
or she is to be influenced by subliminal cues, including hypnosis. Empathy orients people toward other 
people. It allows one to get absorbed in emotional experience and to resonate with others’ experiences, 
including unconsciously mimicking another’s facial expressions or body language. Contagious laughing 
and even contagious yawning are examples of such resonance. 

Empathy is considered essential to social bonding and emotional attachment, but it can also trigger 
“vicarious anxiety” for victims or others who are genuinely distressed. Some psychologists have called 
empathy a “risky strength” because it predisposes us to experience others’ happiness but also their pain.4 
The successful tuning evident in both Facebook experiments is the result of the effective exploitation of 
the natural empathy present in its population of users. 

The Facebook researchers claimed that the results suggested two inferences. First, in a massive and 


engaged population such as Facebook users, even small effects “can have large aggregated 
consequences.” Second, the authors invited readers to imagine what might be accomplished with more- 
significant manipulations and larger experimental populations, noting the importance of their findings for 
“public health.” 

Once again, public outcry was substantial. “If Facebook can tweak emotions and make us vote, what 
else can it do?” the Guardian asked. The Atlantic quoted the study’s editor, who had processed the article 
for publication despite her apparent misgivings.42 She told the magazine that as a private company, 
Facebook did not have to adhere to the legal standards for experimentation required of academic and 
government researchers. 

These legal standards are known as the “Common Rule.” Designed to protect against the abuse of the 
experimenter’s power, these standards must be adhered to by all federally funded research. The Common 
Rule enforces procedures for informed consent, avoidance of harm, debriefing, and transparency, and it is 
administered by panels of scientists, known as “internal review boards,” appointed within every research 
institution. Fiske acknowledged that she had been persuaded by Facebook’s argument that the 
experimental manipulation was an unremarkable extension of the corporation’s standard practice of 
manipulating people’s news feeds. As Fiske recounted, “They said... that Facebook apparently 
manipulates people’s News Feeds all the time.... Who knows what other research they’re doing.”44 In 
other words, Fiske recognized that the experiment was merely an extension of Facebook’s standard 
practices of behavioral modification, which already flourish without sanction. 

Facebook data scientist and principal researcher Adam Kramer was deluged with hundreds of media 
queries, leading him to write on his Facebook page that the corporation really does “care” about its 
emotional impact. One of his coauthors, Cornell’s Jeffrey Hancock, told the New York Times that he didn’t 
realize that manipulating the news feeds, even modestly, would make some people feel violated.” The 
Wall Street Journal reported that the Facebook data science group had run more than 1,000 experiments 
since its inception in 2007 and operated with “few limits” and no internal review board. Writing in the 
Guardian, psychology professor Chris Chambers summarized that “the Facebook study paints a dystopian 
future in which academic researchers escape ethical restriction by teaming up with private companies to 
test increasingly dangerous or harmful interventions.” 

A month after the emotional contagion study’s publication, the editor-in-chief of the Proceedings of 
the National Academy of Sciences, Inder M. Verma, published an “editorial expression of concern” 
regarding the Facebook research. After acknowledging the standard defense that Facebook is technically 
exempt from the “Common Rule,” Verma added, “It is nevertheless a matter of concern that the collection 
of the data by Facebook may have involved practices that were not fully consistent with the principles of 
obtaining informed consent and allowing participants to opt out.”/ 

Among US scholars, University of Maryland law professor James Grimmelmann published the most 
comprehensive argument in favor of holding Facebook and other social media companies accountable to 
the standards represented by the Common Rule. Corporate research is more likely than academic research 
to be compromised by serious conflicts of interests, he reasoned, making common experimental standards 
critical and not something to be left to individual ethical judgment. Grimmelmann imagined “Internal 
Review Board laundering,” in which academics could “circumvent research ethics regulations whenever 
they work just closely enough with industry partners. The exception would swallow the Common Rule.”28 

Despite his conviction on this point, Grimmelmann acknowledged in the final pages of his analysis that 
even the most rigorous imposition of the Common Rule would do little to curb the immense power of a 
company such as Facebook that routinely manipulates user behavior at scale, using means that are 
indecipherable and therefore uncontestable. Like Fiske, Grimmelmann sensed the larger project of 


economies of action just beyond the reach of established law and social norms. 

The journal Nature drew attention with a strongly worded letter defending the Facebook experiment, 
authored by bioethicist Michelle Meyer together with five coauthors and on behalf of twenty-seven other 
ethicists. The letter argued that the need to codify new knowledge about the online environment justifies 
experimentation even when it does not or cannot abide by accepted ethical guidelines for human subjects 
research. But Meyer’s defense turned on a prescient note of warning that “the extreme response to this 
study... could result in such research being done in secret.... If critics think that the manipulation of 
emotional content in this research is sufficiently concerning to merit regulation... then the same concern 
must apply to Facebook’s standard practice....”12 

The experiment’s critics and supporters agreed on little but this: Facebook could easily turn rogue, 
threatening to retreat to secrecy if regulators attempted to intervene in its practices. The academic 
community shared a sense of threat in the face of known facts. Facebook owns an unprecedented means of 
behavior modification that operates covertly, at scale, and in the absence of social or legal mechanisms of 
agreement, contest, and control. Even the most stringent application of the “Common Rule” would be 
unlikely to change these facts. 

As scholars promised to convene panels to consider the ethical issues raised by Facebook research, 
the corporation announced its own plans for better self-regulation. The corporation’s chief technology 
officer, Mike Schroepfer, confessed to being “unprepared” for the public reaction to the emotional 
contagion study, and he admitted that “there are things we should have done differently.” The company’s 
“new framework” for research included clear guidelines, an internal review panel, a capsule on research 
practices incorporated into the company’s famous “boot camp” orientation and training program for new 
hires, and a website to feature published academic research. These self-imposed “regulations” did not 
challenge the fundamental facts of Facebook’s online community as the necessary developmental 
environment and target for the firm’s economies of action. 

A document acquired by the Australian press in May 2017 would eventually reveal this fact. Three 
years after the publication of the contagion study, the Australian broke the story on a confidential twenty- 
three-page Facebook document written by two Facebook executives in 2017 and aimed at the company’s 
Australian and New Zealand advertisers. The report depicted the corporation’s systems for gathering 
“psychological insights” on 6.4 million high school and tertiary students as well as young Australians and 
New Zealanders already in the workforce. The Facebook document detailed the many ways in which the 
corporation uses its stores of behavioral surplus to pinpoint the exact moment at which a young person 
needs a “confidence boost” and is therefore most vulnerable to a specific configuration of advertising 
cues and nudges: “By monitoring posts, pictures, interactions, and Internet activity, Facebook can work 
out when young people feel ‘stressed,’ ‘defeated,’ ‘overwhelmed,’ ‘anxious,’ ‘nervous,’ ‘stupid,’ ‘silly,’ 
‘useless,’ and a ‘failure.’”22 

The report reveals Facebook’s interest in leveraging this affective surplus for the purpose of 
economies of action. It boasts detailed information on “mood shifts” among young people based on 
“internal Facebook data,” and it claims that Facebook’s prediction products can not only “detect 
sentiment” but also predict how emotions are communicated at different points during the week, matching 
each emotional phase with appropriate ad messaging for the maximum probability of guaranteed 
outcomes. “Anticipatory emotions are more likely to be expressed early in the week,” the analysis 
counsels, “while reflective emotions increase on the weekend. Monday-Thursday is about building 
confidence; the weekend is for broadcasting achievements.” 

Facebook publicly denied these practices, but Antonio Garcia-Martinez, a former Facebook product 
manager and author of a useful account of Silicon Valley titled Chaos Monkeys, described in the 


Guardian the routine application of such practices and accused the corporation of “lying through their 
teeth.” He concluded, “The hard reality is that Facebook will never try to limit such use of their data 
unless the public uproar reaches such a crescendo as to be un-mutable.”2! Certainly the public challenge 
to Facebook’s insertion of itself into the emotional lives of its users, as expressed in the contagion study, 
and its pledge to self-regulate did not quell its commercial interest in users’ emotions or the corporation’s 
compulsion to systematically exploit that knowledge on behalf of and in collaboration with its customers. 
It did not, because it cannot, not as long as the company’s revenues are bound to economies of action 
under the authority of the prediction imperative. 

Facebook’s persistence warns us again of the dispossession cycle’s stubborn march. Facebook had 
publicly acknowledged and apologized for its overt experimental incursions into behavior modification 
and emotional manipulation, and it promised adaptations to curb or mitigate these practices. Meanwhile, 
a new threshold of intimate life had been breached. Facebook’s potential mastery of emotional 
manipulation became discussable and even taken for granted as habituation set in. From Princeton’s 
Fiske to critic Grimmelmann and supporter Meyer, the experts believed that if Facebook’s activities were 
to be forced into a new regulatory regime, the corporation would merely continue in secret. The 
Australian documents opened one door on these covert practices, suggesting the completion of the cycle 
with the redirection of action into clandestine zones protected by opacity and indecipherability, just as 
these scholars had anticipated. 

Facebook’s political mobilization experimenters discovered that they could manipulate users’ 
vulnerabilities to social influence in order to create a motivational condition (“I want to be like my 
friends”) that increases the probability that a relevant priming message—the “I Voted” button—will 
produce action. The emotional contagion study exploited the same underlying social influence orientation. 
In this case, Facebook planted subliminal cues in the form of positive or negative affective language, 
which combined with the motivational state triggered by social comparison—“I want to be like my 
friends”—to produce a measurable, if weak, contagion effect. Finally, the Australian ad-targeting 
document points to the seriousness and complexity of the backstage effort to strengthen this effect by 
specifying motivational conditions at a granular level. It reveals not only the scale and scope of 
Facebook’s behavioral surplus but also the corporation’s interest in leveraging its surplus to precisely 
determine the ebb and flow of a user’s predisposition for real-time targeting by the branded cues that are 
most likely to achieve guaranteed outcomes. 

Facebook’s experimental success demonstrates that tuning through suggestion can be an effective form 
of telestimulation at scale. The evasion of individual and group awareness was critical to Facebook’s 
behavior-modification success, just as MacKay had stipulated. The first paragraph of the research article 
on emotional contagion celebrates this evasion: “Emotional states can be transferred to others via 
emotional contagion, leading people to experience the same emotions without their awareness.” Nor do 
the young adults of Australia’s great cities suspect that the precise measure of their fears and fantasies is 
exploited for commercial result at the hour and moment of their greatest vulnerability. 

This evasion is neither accidental nor incidental, but actually essential to the structure of the whole 
surveillance capitalist project. Individual awareness is the enemy of telestimulation because it is the 
necessary condition for the mobilization of cognitive and existential resources. There is no autonomous 
judgment without awareness. Agreement and disagreement, participation and withdrawal, resistance or 
collaboration: none of these self-regulating choices can exist without awareness. 

A rich and flourishing research literature illuminates the antecedents, conditions, consequences, and 
challenges of human self-regulation as a universal need. The capacity for self-determination is understood 
as an essential foundation for many of the behaviors that we associate with critical capabilities such as 


empathy, volition, reflection, personal development, authenticity, integrity, learning, goal accomplishment, 
impulse control, creativity, and the sustenance of intimate enduring relationships. “Implicit in this process 
is a self that sets goals and standards, is aware of its own thoughts and behaviors, and has the capacity to 
change them,” write Ohio State University professor Dylan Wagner and Dartmouth professor Todd 
Heatherton in an essay about the centrality of self-awareness to self-determination: “Indeed, some 
theorists have suggested that the primary purpose of self awareness is to enable self-regulation.” Every 
threat to human autonomy begins with an assault on awareness, “tearing down our capacity to regulate our 
thoughts, emotions, and desires.”22 

The salience of self-awareness as a bulwark against self-regulatory failure is also underscored in the 
work of two Cambridge University researchers who developed a scale to measure a person’s 
“susceptibility to persuasion.” They found that the single most important determinant of one’s ability to 
resist persuasion is what they call “the ability to premeditate.”22 This means that people who harness self- 
awareness to think through the consequences of their actions are more disposed to chart their own course 
and are significantly less vulnerable to persuasion techniques. Self-awareness also figures in the second- 
highest-ranking factor on their scale: commitment. People who are consciously committed to a course of 
action or set of principles are less likely to be persuaded to do something that violates that commitment. 

We have seen already that democracy threatens surveillance revenues. Facebook’s practices suggest 
an equally disturbing conclusion: human consciousness itself is a threat to surveillance revenues, as 
awareness endangers the larger project of behavior modification. Philosophers recognize “self- 
regulation,” “self-determination,” and “autonomy” as “freedom of will.” The word autonomy derives 
from the Greek and literally means “regulation by the self.” It stands in contrast to heteronomy, which 
means “regulation by others.” The competitive necessity of economies of action means that surveillance 
capitalists must use all means available to supplant autonomous action with heteronomous action. 

In one sense there is nothing remarkable in observing that capitalists would prefer individuals who 
agree to work and consume in ways that most advantage capital. We need only to consider the ravages of 
the subprime mortgage industry that helped trigger the great financial crisis of 2008 or the daily insults to 
human autonomy at the hands of countless industries from airlines to insurance for plentiful examples of 
this plain fact. 

However, it would be dangerous to nurse the notion that today’s surveillance capitalists simply 
represent more of the same. This structural requirement of economies of action turns the means of 
behavioral modification into an engine of growth. At no other time in history have private corporations of 
unprecedented wealth and power enjoyed the free exercise of economies of action supported by a 
pervasive global architecture of ubiquitous computational knowledge and control constructed and 
maintained by all the advanced scientific know-how that money can buy. 

Most pointedly, Facebook’s declaration of experimental authority claims surveillance capitalists’ 
prerogatives over the future course of others’ behavior. In declaring the right to modify human action 
secretly and for profit, surveillance capitalism effectively exiles us from our own behavior, shifting the 
locus of control over the future tense from “I will” to “You will.” Each one of us may follow a distinct 
path, but economies of action ensure that the path is already shaped by surveillance capitalism’s economic 
imperatives. The struggle for power and control in society is no longer associated with the hidden facts of 
class and its relationship to production but rather by the hidden facts of automated engineered behavior 
modification. 


Il. Pokémon Go! Do! 


It had been a particularly grueling July afternoon in 2016. David had directed hours of contentious 
insurance testimony in a dusty New Jersey courtroom, where a power surge the night before had knocked 
out the building’s fragile air-conditioning system. Then the fitful Friday commute home was cursed by a 
single car disabled by the heat that turned the once-hopeful flow of traffic into sludge. Finally home, he 
slid the car into his garage and made a beeline for the side door that opened to the laundry room and 
kitchen beyond. The cool air hit him like a dive into the ocean, and for the first time all day he took a deep 
breath. A note on the table said his wife would be back in a few minutes. He gulped down some water, 
made himself a drink, and climbed the stairs, heading for a long shower. 

The doorbell rang just as the warm water hit his aching back muscles. Had she forgotten her key? 
Shower interrupted, he threw on a tee and shorts and ran downstairs, opening the front door to a couple of 
teenagers waving their cell phones in his face. “Hey, you’ve got a Pokémon in your backyard. It’s ours! 
Okay if we go back there and catch it?” 

“A what?” He had no idea what they were talking about, but he was about to get educated. 

David’s doorbell rang four more times that evening: perfect strangers eager for access to his yard and 
disgruntled when he asked them to leave. Throughout the days and evenings that followed, knots of 
Pokémon seekers formed on his front lawn, some of them young and others long past that excuse. They 
held up their phones, pointing and shouting as they scanned his house and garden for the “augmented- 
reality” creatures. Looking at this small slice of world through their phones, they could see their Pokémon 
prey but only at the expense of everything else. They could not see a family’s home or the boundaries of 
civility that made it a sanctuary for the man and woman who lived there. Instead, the game seized the 
house and the world around it, reinterpreting all of it ina vast equivalency of GPS coordinates. Here was 
a new kind of commercial assertion: a for-profit declaration of eminent domain in which reality is recast 
as an unbounded expanse of blank spaces to be sweated for others’ enrichment. David wondered, When 
will this end? What gives them the right? Whom do I call to make this stop? 

Without knowing it, he had been yanked from his shower to join the villagers in Broughton, England, 
who had taken to their streets in 2009 protesting the invasion of Google’s Street View camera cars. Like 
them, he had been abruptly thrust into contest with surveillance capitalism’s economic imperatives, and 
like them he would soon understand that there was no number to call, no 911 to urgently inform the 
appropriate authorities that a dreadful mistake had blossomed on his lawn. 

Back in 2009, as we saw in Chapter 5, Google Maps product vice president and Street View boss 
John Hanke ignored the Broughton protestors, insisting that only he and Google knew what was best, not 
just for Broughton but for all people. Now here was Hanke again at surveillance capitalism’s next 
frontier, this time as the founder of the company behind Pokémon Go, Niantic Labs. Hanke, you may 
recall, nursed an abiding determination to own the world by mapping it. He had founded Keyhole, the 
satellite mapping startup funded by the CIA and later acquired by Google and rechristened as Google 
Earth. At Google, he was a vice president for Google Maps and a principal in its controversial push to 
commandeer public and private space through its Street View project. 

Hanke recounts that Pokémon Go was born out of Google Maps, which also supplied most of the 
game’s original development team.~4 Indeed, Street View’s mystery engineer, Marius Milner, had joined 
Hanke in this new phase of incursion. By 2010, Hanke had set up his own launch pad, Niantic Labs, 
inside the Google mother ship. His aim was the development of “parallel reality” games that would track 
and herd people through the very territories that Street View had so audaciously claimed for its maps. In 
2015, following the establishment of the Alphabet corporate structure and well after the development of 
Pokémon Go, Niantic Labs was formally established as an independent company with $30 million in 
funding from Google, Nintendo (the Japanese company that originally hosted Pokémon on its “Game Boy” 


devices in the late 1990s), and the Pokémon Company.” 

Hanke had long recognized the power of the game format as a means to achieve economies of action. 
While still at Google he told an interviewer, “More than 80% of people who own a mobile device claim 
that they play games on their device... games are often the number 1 or number 2 activity... so for 
Android as an operative system, but also for Google, we think it’s important for us to innovate and to be a 
leader in... the future of mobile gaming.”© 

It is worth noting that Hanke chose to name his group after a nineteenth-century merchant sailing vessel 
undone by greed. The Niantic had been sold and repurposed for the more lucrative whaling trade when it 
set sail for San Francisco and the northern Pacific whaling grounds in 1849. The ship’s captain made an 
unplanned stop in Panama to board hundreds of pilgrims bound for the California Gold Rush, all of them 
eager to pay top dollar for cramped, smelly quarters on the whaler. The captain’s avarice proved fatal to 
the ship’s prospects when those passengers infected the ship’s crew with gold fever. The sailors 
abandoned captain and vessel upon docking in San Francisco, heading instead for gold country. Unable to 
continue the journey, the captain was forced to sell the ship for a pittance, leaving it wedged deep in the 
sandy shallows at the foot of Clay and Montgomery streets. In 2016 Hanke took up the quest of that 
rebellious crew. His Niantic was bound for a new century’s gold rush at the frontier of the prediction 
imperative’s next wave of conquest: economies of action. 

Hanke’s Pokémon Go launched in July 2016 as a different answer to the question confronting the 
engineers and scientists shaping the surveillance capitalist project: how can human behavior be actuated 
quickly and at scale, while driving it toward guaranteed outcomes? At its zenith in the summer of 2016, 
Pokémon Go was a surveillance capitalist’s dream come true, fusing scale, scope, and actuation; yielding 
continuous sources of behavioral surplus; and providing fresh data to elaborate the mapping of interior, 
exterior, public, and private spaces. Most important, it provided a living laboratory for telestimulation at 
scale as the game’s owners learned how to automatically condition and herd collective behavior, 
directing it toward real-time constellations of behavioral futures markets, with all of this accomplished 
just beyond the rim of individual awareness. In Hanke’s approach, economies of action would be 
achieved through the dynamics of a game. 

Niantic designed the new game to be “played” in the real world, not on a screen. The idea is that 
players should be “going outside” for “adventures on foot” in the open spaces of cities, towns, and 
suburbs.22 The game relies on “augmented reality” and is structured like a treasure hunt. Once you 
download the app from Niantic, you use GPS and your smartphone camera to hunt virtual creatures called 
Pokémon. The figures appear on your smartphone screen as if they are located beside you in your real-life 
surroundings: an unsuspecting man’s backyard, a city street, a pizzeria, a park, a drugstore. Captured 
Pokémon are rewarded with game currencies, candies, and stardust, and are employed to battle other 
users. The ultimate goal is to capture a comprehensive array of the 151 Pokémon, but along the way 
players earn “experience points,” rising to successive levels of expertise. At level five, players can join 
one of three teams to battle Pokémon at designated sites referred to as “gyms.” 

The ramp-up had begun years earlier with Ingress, Niantic’s first mobile game designed for real- 
world play. Released in 2012, Ingress was a precursor and test bed for the capabilities and methods that 
would define Pokémon Go. The game drove its users through their cities and towns to find and control 
designated “portals” and capture “territory” as the game masters relied on GPS to track users’ movements 
and map the territories through which they roamed. 

Hanke reflected on what he and his team had learned from Ingress. Most important was the Niantic 
team’s “surprise” as they observed how much “the behavior of the players changes.”22 Hanke grasped that 
the seeds of behavior modification were planted in the game’s rules and social dynamic: “If you want to 


turn the world into your game board, the places you want people to interact with have to have certain 
characteristics.... There should be a reason for the player to go there.... The game is enabling them and 
nudging you to have those interactions.”22 One user whose Ingress name was “Spottiswoode” provides an 
example: “As I cycle home, I stop near a location I’d scouted out previously, one with a weak enemy 
portal. I attack, using built-up XM (“exotic matter’) to destroy the enemy infrastructure.... On Ingress’s 
built-in chat client, a player called Igashu praises my handiwork. ‘Good job, Spottiswoode,’ he says. I 
feel proud and move on, plotting my next assault upon the enemy’s portals.”22 According to Hanke, 
Pokémon Go would be designed to leverage what the team now understood as the key sources of 
motivation that induce players to change their behavior: a social gaming community based on real-world 
action.2! 

All games circumscribe behavior with rules, rewarding some forms of action and punishing others, 
and Niantic is not the first to employ the structure of a game as a means of effecting behavior change in its 
players. Indeed, “gamification” as an approach to behavioral engineering is a subject of intense interest 
that has produced a robust academic and popular literature.2* According to Wharton professor Kevin 
Werbach, games include three tiers of action. At the highest level are the “dynamics” that drive the 
motivational energy of the game. These can be emotions aroused by competition or frustration, a 
compelling narrative, a structure of progression that creates the experience of development toward a 
higher goal, or relationships that produce feelings such as team spirit or aggression. Next are the 
“mechanics.” These are the procedural building blocks that drive the action and also build engagement. 
For example, a game may be structured as a competition or a solo challenge, as turn taking and 
cooperation, as transactions and winner take all, as team sport or individual conquest. Finally, there are 
the game “components” that operationalize the mechanics. These are the most-visible aspects of a game: 
points to represent progress, quests laid out as predefined challenges, “badges” to represent 
achievements, “leader boards” to visually display all players’ progress, “boss fights” to mark the 
culmination of a level, and so forth.=2 

Most research on games concludes that these structures can be effective at motivating action, and 
researchers generally predict that games will increasingly be used as the methodology of choice to change 
individual behavior.“ In practice, this has meant that the power of games to change behavior is 
shamelessly instrumentalized as gamification spreads to thousands of situations in which a company 
merely wants to tune, herd, and condition the behavior of its customers or employees toward its own 
objectives. Typically, this involves importing a few components, such as reward points and levels of 
advancement, in order to engineer behaviors that serve the company’s immediate interests, with programs 
such as customer loyalty schemes or internal sales competitions. One analyst compiled a survey of more 
than ninety such “gamification cases,” complete with return-on-investment statistics.22 Ian Bogost, a 
professor of interactive computing at Georgia Tech and a digital culture observer, insists that these 
systems should be called “exploitationware” rather than games because their sole aim is behavior 
manipulation and modification. 

Pokémon Go takes these capabilities in a wholly new direction, running game players through the real 
world, but not for the sake of the game they think they are playing. Hanke’s unique genius is to point the 
game’s behavior-modification efforts toward a target that occupies an unexplored zone beyond the 
boundaries of players’ awareness. It aims to shape behavior in an even larger game of surveillance 
capitalism. 

Pokémon Go was first unveiled to the Wall Street Journal in September 2015, shortly after Niantic’s 
spin-off from Google. The game masters told the reporter that the game would not include ads. Instead, 
revenues would accrue from “microtransactions,” presumably in-game purchases of virtual paraphernalia, 


although Niantic “declined to say” exactly what would be for sale. Niantic also promised a location- 
tracking bracelet that “vibrates and lights up” when a person approaches a Pokémon. It was clear that 
Pokémon Go would at least be a fresh source of surplus for refining and expanding the maps upon which 
the game depended. 

Released in the US, Australia, and New Zealand on July 6, 2016, Pokémon Go became the most 
downloaded and highest-grossing app in the US within only a week, quickly achieving as many active 
Android users as Twitter. More than 60 percent of the app’s downloads were in daily use, and by July 8 
that translated into a daily average of about 43.5 minutes per user.22 With Niantic’s servers groaning under 
the strain, the game’s European rollout was delayed until July 13. By that time, however, Niantic had 
proved the value of its approach to economies of action, demonstrating unprecedented effectiveness in 
traversing that last tortured mile to guaranteed outcomes. 

The unprecedented pattern was faintly discernible within days of the game’s launch. A Virginia bar 
offered a discount to a Pokémon Go team; a tea shop in San Francisco offered a “buy one get one free” to 
the game’s players.°2 The owner of a pizza bar in Queens, New York, paid about $10 for “Lure Modules,” 
a bit of virtual game paraphernalia intended to attract Pokémon to a specific location, successfully 
producing virtual creatures on bar stools and in bathroom stalls. During the first weekend of game play, 
the bar’s food and drink sales shot up by 30 percent and later were reported to be 70 percent above 
average. Bloomberg reporters gushed that the game had achieved the retailers’ elusive dream of using 
location tracking to drive foot traffic: “It’s easy to imagine a developer selling ads within the game world 
to local merchants, or even auctioning off the promise to turn specific shops and restaurants into 
destinations for players.”42 Hanke hinted to the New York Times that these real-world, real-time markets 
had been the plan all along. “Niantic has cut deals like that for Ingress,” the paper reported, “and Mr. 
Hanke said the company would announce sponsored locations for Pokémon Go in the future.” 

The future came quickly. Within a week the basic elements of surveillance capitalism’s logic of 
accumulation were in place and were heralded as brilliant. As Hanke explained, “The game relies on a 
lot of modern cell phone and data technology to power the augmented reality, but that traffic generated by 
the game also changes what happens in the real world.”“4 By July 12, the Financial Times exulted that 
“speculation has surged over the game’s future power as a cash cow to retailers and other cravers of 
footfall.” Nintendo shares were up 52 percent, adding $10.2 billion to its market capitalization.“ 

Earlier promises that the game would not serve ads turned out to be a technical claim that required 
careful parsing. In fact, the surveillance-based logic of online advertising had not disappeared. Rather, it 
had morphed into its physical-world mirror image, just as Sidewalk Labs’ Dan Doctoroff had imagined 
for the “Google city,” a precise extension of the methods and purposes honed in the online world but now 
amplified in “reality” under pressure from the prediction imperative (see Chapter 7). 

By July 13, Hanke admitted to the Financial Times that in addition to “in-app payments” for game kit, 
“there is a second component to our business model at Niantic, which is the concept of sponsored 
locations.” He explained that this new revenue stream had always been in the plan, noting that companies 
will “pay us to be locations within the virtual game board—the premise being that it is an inducement that 
drives foot traffic.” These sponsors, Hanke explained, would be charged on a “cost per visit” basis, 
similar to the “cost per click” used in Google’s search advertising.“ 

The notion of “sponsored locations” is a euphemism for Niantic’s behavioral futures markets, ground 
zero in Hanke’s new gold rush. The elements and dynamics of the game, combined with its novel 
augmented-reality technology, operate to herd populations of game players through the real-world 
monetization checkpoints constituted by the game’s actual customers: the entities who pay to play on the 
real-world game board, lured by the promise of guaranteed outcomes. 


For a while it seemed that everyone was making money. Niantic inked a deal with McDonald’s to 
drive game users to its 30,000 Japanese outlets. A British mall owner commissioned “recharging teams” 
to roam his malls with portable rechargers for game users. Starbucks announced that it would “join in 
with the fun,” with 12,000 of its US stores becoming official “Pokéstops” or “gyms,” along with a new 
“Pokémon Go Frappuccino... the perfect treat for any Pokémon trainer on the go.” Another deal with 
Sprint would convert 10,500 Sprint retail and service outlets into Pokémon hubs. The music streaming 
company Spotify reported a tripling of Pokémon-related music sales. A UK insurance company offered 
special coverage for mobile phones, warning, “Don’t let accidental damage get in the way of catching 
them all.” Disney admitted that it was disappointed with its own strategies for “the blending of physical 
and digital to create new kinds of connected play experiences” and planned to transform its mammoth toy 
business “in a direction similar to Pokémon Go.” 

The zeal for Pokémon Go gradually diminished, but the impact of Hanke’s accomplishments is 
indelible. “We’ve only just scratched the surface,” Hanke told a crowd of fans.“© The game had 
demonstrated that it was possible to achieve economies of action on a global scale while simultaneously 
directing specific individual actions toward precise local market opportunities where high bidders enjoy 
an ever-closer approximation of guaranteed outcomes. 

Niantic’s distinctive accomplishment was to manage gamification as a way to guarantee outcomes for 
its actual customers: companies participating in the behavioral futures markets that it establishes and 
hosts. Hanke’s game proved that surveillance capitalism could operate in the real world much as it does 
in the virtual one, using its unilateral knowledge (scale and scope) to shape your behavior now (action) in 
order to more accurately predict your behavior later. The logical inference is that real-world revenues 
will increase in proportion to the company’s ability to match persons with locations, just as Google 
learned to wield surplus as a means of targeting online ads to specific individuals. 

These requirements suggest that Niantic would conduct its operations in ways that establish substantial 
surplus supply chains aimed at scale and scope. Indeed, the company’s “surveillance policy” signals its 
demand for behavioral data in excess of what is reasonable for effective game operations. Just six days 
after the game’s release in July 2016, BuzzFeed reporter Joseph Bernstein advised Pokémon users to 
check how much data the app was collecting from their phones. According to his analysis, “Like most 
apps that work with the GPS in your smartphone, Pokémon Go can tell a lot of things about you based on 
your movement as you play: where you go, when you went there, how you got there, how long you stayed, 
and who else was there. And, like many developers who build those apps, Niantic keeps that 
information.” Whereas other location-based apps might collect similar data, Bernstein concluded that 
“Pokémon Go’s incredibly granular, block-by-block map data, combined with its surging popularity, may 
soon make it one of, if not the most, detailed location-based social graphs ever compiled.”“2 

The industry news site TechCrunch raised similar concerns regarding the game’s data-collection 
practices, questioning “the long list of permissions the app requires.” Those permissions included the 
camera, yes, but also permission to “read your contacts” and “find accounts on device.” Niantic’s 
“surveillance policy” notes that it may share “aggregated information and non-identifying information 
with third parties for research and analysis, demographic profiling, and other similar purposes.” 
TechCrunch noted the game’s “precise location tracking” and “ability to perform audio fingerprinting” 
through its access to your camera and microphone, concluding, “So it’s prudent to expect some of your 
location data to end up in Google’s hands.”“’ The Electronic Privacy Information Center noted in a letter 
of complaint to the Federal Trade Commission that Niantic had failed to provide compelling reasons for 
the “scope” of the information that it routinely gathers from users’ phones and Google profiles. Nor had it 
set limits on how long it would retain, use, or share location data. As the letter concluded, “There is no 


evidence that Niantic’s collection and retention of location data is necessary to the function of the game or 
otherwise provides a benefit to consumers that outweighs the privacy and safety harms it creates.” 

By mid-July 2016, Niantic received a letter from US Senator Al Franken querying the company’s 
privacy practices.22 Niantic’s late-August response is instructive, a marvel of misdirection and secrecy 
that focuses on the game’s mechanics and discloses nothing about its business model or the more 
comprehensive logic of accumulation behind the model: “Pokémon Go has already been praised by public 
health officials, teachers, mental health workers, parents, park officials, and ordinary citizens around the 
world as an app that promotes healthy play and discovery.” Though acknowledging the range of data it 
collects as a condition of play—tlocation services, photos, media, files, camera, contacts, and network 
provider data—Niantic insists that data are used “to provide and improve” its services. However, it does 
not acknowledge that its services operate on two levels: game services for players and prediction 
services for Niantic’s customers. The company concedes that it uses third-party services, including 
Google’s, to “collect and interpret data,” but it is careful to sidestep the aims of those analyses.2! 

The seven-page letter mentions “sponsored locations” only once, noting that sponsors receive reports 
about visits and game actions. There is no reference to “cost per visit” or the surplus that will be required 
to drive that metric, in the same way that Google’s “cost per click” depended upon behavioral surplus 
drawn from online activity. Niantic’s self-presentation carefully conceals its objectives in the design and 
development of economies of action that drive real-world, real-time behavior toward Niantic’s 
behavioral futures markets. 

The genius of Pokémon Go was to transform the game you see into a higher-order game of surveillance 
capitalism, a game about a game. The players who took the city as their board, roaming its parks and 
pizzerias, unwittingly constituted a wholly different kind of human game board for this second and more 
consequential game. The players in this other real game could not be found in the clot of enthusiasts 
waving their phones at the edge of David’s lawn. In the real game, prediction products take the form of 
protocols that impose forms of telestimulation intended to prod and herd people across real-world 
terrains to spend their real-world money in the real-world commercial establishments of Niantic’s flesh- 
and-blood behavioral futures markets. 

Niantic itself is like a tiny probe rising from the immensity of Google’s mapping capabilities, surplus 
flows, means of production, and vast server farms as it constructs and tests the prototype of a global 
means of behavior modification owned and operated by surveillance capitalism. Niantic discovered that 
in the rapture of engaging competitive social play, the dreaded friction of individual will voluntarily gives 
way to game protocols that set the conditions for “natural selection.” In this way the game automatically 
elicits and breeds the specific behaviors sought by the high rollers in Niantic’s behavioral futures 
markets. With this second game board in motion, the players in the real game vie for proximity to the 
wake of cash that follows each smiling member of the herd. 

In the end we recognize that the probe was designed to explore the next frontier: the means of 
behavioral modification. The game about the game is, in fact, an experimental facsimile of surveillance 
capitalism’s design for our future. It follows the prediction imperative to its logical conclusion, in which 
data about us in scale and scope combine with actuation mechanisms that align our behavior with a new 
market cosmos. All the flows of surplus from all the spaces, all the things, all the bodies, all the laughter, 
and all the tears are finally aimed at this triumph of certain outcomes and the revenue that it can unleash. 


IV. What Were the Means of Behavioral Modification? 


The new global means of behavioral modification that we see under construction at Facebook and Niantic 
represent a new regressive age of autonomous capital and heteronomous individuals, when the very 
possibilities of democratic flourishing and human fulfillment depend upon the reverse. This 
unprecedented state of affairs rises above debates about the Common Rule. It goes to the heart of our 
allegiance to the ideals of a democratic society, with full knowledge of the challenges that burden those 
ideals. 

What has been forgotten here is that the Common Rule was the product of a similar challenge to 
principles of individual autonomy and democratic fidelity. It was one result of a deeply contested struggle 
in which democratically minded public officials joined with social activists, scholars, and litigators to 
resist the design, development, and deployment of behavioral modification as a mode of governmental 
power. It was not long ago that US society mobilized to resist, regulate, and control the means of 
behavioral modification, and it is this history that we can now draw upon to rediscover our bearings and 
rouse our awareness. 

In 1971 the Senate Subcommittee on Constitutional Rights, led by North Carolina Senator Sam Ervin 
and including luminaries from across the political spectrum such as Edward Kennedy, Birch Bayh, Robert 
Byrd, and Strom Thurmond, undertook what would become a multiyear investigation into “a variety of 
programs designed to predict, control, and modify human behavior.” Ervin was a conservative Democrat 
and constitutional expert who became an unlikely civil liberties hero for his defense of democracy during 
the Watergate crisis as chair of the Senate Watergate Committee. In this case the Subcommittee on 
Constitutional Rights would subject the principles and applications of behavior modification to intense 
constitutional scrutiny for the first time, questioning and ultimately rejecting the use of behavioral 
modification as an extension of state power. 

The Senate investigation was triggered by a growing sense of public alarm at the spread of 
psychological techniques for behavior control. There were many points of origin, but most salient was the 
influence of the cold war and the range of psychological techniques and programs for behavior 
modification that it had bred. The Korean War had publicized communist “brainwashing” techniques that, 
according to then newly appointed CIA Director Allen Dulles, had reduced US prisoners of war to a state 
of robotic passivity, in which the victim’s brain “becomes a phonograph playing a disc put on its spindle 
by an outside genius over which it has no control.”22 America’s enemies appeared to be on the verge of 
mastering the art and science of “mind control” with psychological and pharmacological methods 
unknown to the US military. There were reports of Chinese and Soviet achievements in the remote 
alteration of a subject’s mental capacities and the elimination of his “free will.”2? Dulles committed the 
agency to rapid research in and development of “mind control” capabilities, from “de-patterning” and 
“rewiring” an individual to shaping an entire country’s attitudes and actions.“ 

Thus began a morbidly fascinating and often bizarre chapter in the history of American spy craft.22 
Much of the new work was conducted in the context of the CIA’s highly classified MKUltra project, 
which was tasked with “research and development of chemical, biological, and radiological materials 
capable for employment in clandestine operations to control human behavior.” According to testimony in 
the 1975 Senate investigation of covert CIA Foreign and Military Intelligence operations, a 1963 
Inspector General’s report on MKUltra noted several reasons for the program’s secrecy, but chief among 
them was the fact that behavior modification was seen as illegitimate. “Research in the manipulation of 
human behavior is considered by many authorities in medicine and related fields to be professionally 
unethical, therefore the reputation of professional participants in the MKUltra program are on occasion in 
jeopardy,” the report began. It also noted that many of the program’s activities were illegal, violated the 


rights and interests of US citizens, and would alienate public opinion.° 


Key to our interests is the growth and elaboration of behavioral modification as an extension of 
political power. To this end, CIA “demand” summoned an ever-more-audacious supply of behavioral- 
modification research and practical applications from academic psychologists. Scientists from the fields 
of medicine and psychology set out to demystify Chinese brainwashing techniques, reinterpreting them 
through the established frameworks of behavior modification. 

Their research concluded that “mind control” was better understood as a complex system of 
conditioning based on unpredictable schedules of reinforcement, consistent with B. F. Skinner’s important 
discoveries on operant conditioning. According to Harvard historian Rebecca Lemov, the “mind control” 
researchers had a powerful effect on the CIA and other branches of the military. The notion that “human 
material was changeable”—that one’s personality, identity, awareness, and capacity for self-determining 
behavior could be crushed, eliminated, and replaced by external control—incited a new sense of panic 
and vulnerability: “If indeed the world was rife with threats to the inner as much as the outer man, then 
experts in these realms were needed more than ever. Many good and well-meaning professors—self- 
described or de facto human engineers—participated in the CIA’s programs to bring about slow or rapid 
change in the minds and behavior of people.”=2 

By the time the senators on the Constitutional Rights Subcommittee convened in 1971, the migration of 
behavior-modification practices from military to civilian applications was well underway. Behavior- 
modification techniques had dispersed from government-funded (typically CIA) psychology labs and 
military psyops to a range of institutional applications, each driven by a mission to reengineer the 
defective personalities of captive individuals in settings that offered “total control” or close to it, 
including prisons, psychiatric wards, classrooms, institutions for the mentally challenged, schools for the 
autistic, and factories. 

The subcommittee was emboldened to act when public concern bubbled over into outrage at the 
mainstreaming of these behavior-modification programs. Historian of psychology Alexandra Rutherford 
observes that Skinnerian behavior-modification practices expanded rapidly during the 1960s and 1970s, 
achieving some “remarkable successes” but also exposing practitioners to the scrutiny of an often-hostile 
public. A number of journalistic accounts raised alarms about the zealousness with which behavior- 
modification techniques were applied and the sense that they debased their subjects, violated ethical 
considerations, and infringed on fundamental civil liberties.°2 

Another factor was the 1971 publication of B. F. Skinner’s incendiary social meditation Beyond 
Freedom & Dignity. Skinner prescribed a future based on behavioral control, rejecting the very idea of 
freedom (as well as every tenet of a liberal society) and cast the notion of human dignity as an accident of 
self-serving narcissism. Skinner imagined a pervasive “technology of behavior” that would one day 
enable the application of behavior-modification methods across entire human populations. 

The ensuing storm of controversy made Beyond Freedom & Dignity an international best seller. 
“Skinner’s science of human behavior, being quite vacuous, is as congenial to the libertarian as to the 
fascist,” wrote Noam Chomsky in a widely read review of the book. “It would be not absurd but 
grotesque to argue that since circumstances can be arranged under which behavior is quite predictable— 
as in a prison, for example, or... concentration camp.... therefore there need be no concern for the 
freedom and dignity of ‘autonomous man.’”22 (In the mid-1970s graduate department at Harvard where I 
studied and Skinner professed, many students referred to the book as Toward Slavery and Humiliation.) 

From the first lines of the preface of the subcommittee’s 1974 report, authored by Senator Ervin, it 
should be evident to any twenty-first-century captive of surveillance capitalism that US society has 
undergone a social discontinuity more profound than the mere passage of decades suggests. It is worth 
reading Ervin’s own words, to grasp the passion with which he located the subcommittee’s work at the 


heart of the Enlightenment project, pledging to defend the liberal ideals of freedom and dignity: 


When the founding fathers established our constitutional system of government, they based it on their fundamental belief in the 
sanctity of the individual.... They understood that self-determination is the source of individuality, and individuality is the mainstay of 
freedom.... Recently, however, technology has begun to develop new methods of behavior control capable of altering not just an 
individual’s actions but his very personality and manner of thinking... the behavioral technology being developed in the United States 
today touches upon the most basic sources of individuality and the very core of personal freedom... the most serious threat... is the 
power this technology gives one man to impose his views and values on another.... Concepts of freedom, privacy and self- 
determination inherently conflict with programs designed to control not just physical freedom, but the source of free thought as well. 
... The question becomes even more acute when these programs are conducted, as they are today, in the absence of strict controls. 
As disturbing as behavior modification may be on a theoretical level, the unchecked growth of the practical technology of behavior 


control is cause for even greater concern.” 


The report’s critique of behavior modification has unique relevance for our time. It begins by asking a 
question that we must also ask: “How did they get away with it?” Their answer invokes the 
“exceptionalism” of that era. Just as surveillance capitalism was initially able to root and flourish under 
the protection of a so-called “war against terror” and the compulsion for certainty that it stirred, in the 
middle of the twentieth century the means of behavioral modification migrated from the lab to the world at 
large primarily under the cover of cold-war anxieties. Later, the behavior-change professionals of the 
1960s and 1970s were summoned into civilian practice by a society turned fearful after years of urban 
riots, political protests, and rising levels of crime and “delinquency.” The senators reasoned that calls for 
“law and order” had motivated the search for “immediate and efficient means to control violence and 
other forms of anti-social behavior. The interest in controlling violence replaced more time-consuming 
attempts to understand its sources.” 

With so many behavior-modification programs aimed at involuntary populations in state prisons and 
mental institutions, the senators recognized that the means of behavioral modification had to be reckoned 
with as a form of state power and questioned the government’s constitutional right to “control” citizens’ 
behavior and mentation. In its survey of government agencies, the subcommittee found “a wide variety of 
behavior modification techniques... presently employed in the United States under the auspices of the 
federal government” and observed that “with the rapid proliferation of behavior modification techniques, 
it is all the more disturbing that few real efforts have been made to consider the basic issues of individual 
freedom involved and... fundamental conflicts between individual rights and behavior technology.”= 

The senators reserved their most vivid rebukes for what they considered as the two most extreme and 
pernicious behavior-modification techniques. The first was psychosurgery. The second was 
“electrophysiology,” defined as “the use of mechanical devices to control various aspects of human 
behavior.” The report notes with special horror the example of “devices” designed to be worn by a 
subject “constantly to monitor and control his behavior through a computer” and to “prevent a suspected 
behavior from occurring.” 

The First Amendment, the subcommittee argued, “must equally protect the individual’s right to 
generate ideas,” and the right to privacy should protect citizens from intrusions into their thoughts, 
behavior, personality, and identity lest these concepts “become meaningless.” It was in this context that 
Skinnerian behavioral engineering was singled out for critical examination: “A major segment of the 
emerging behavior control technology is concerned with conditioning, through which various forms of 
persuasion are used to stimulate certain types of behaviors while suppressing others.”® 

In anticipation of future gamification techniques as means of behavioral modification, the 


subcommittee report also noted with apprehension more “benign” approaches that relied on “positive 
reinforcement,” from “gold-star incentives” to elaborate reward systems, in order “to restructure 
personality through artificially applied techniques.” The generalized obsession with controlling violence 
had also produced methods of “behavior prediction” that “raise profound questions with respect to due 
process, privacy, and individual liberties.” A psychologist writing in the American Psychological 
Association’s journal Monitor in 1974 sounded the alert, warning colleagues who touted their ability to 
“control behavior” that they were now “viewed with increasing suspicion, if not revulsion, and threatened 
with restriction... The social control of behavior control is underway.” 

The subcommittee’s work had enduring consequences. Not only did prisoners’ and patients’ rights 
groups gain momentum in their efforts to end the behavioral oppression suffered in public institutions, but 
psychologists also began to discuss the need to professionalize their discipline with clear ethical 
standards, accreditation procedures, training programs, and career ladders.“ The National Research Act, 
passed in 1974, stipulated the creation of institutional review boards and laid the foundation for the 
evolution and institutionalization of the Common Rule for the ethical treatment of human subjects, from 
which Facebook famously held itself exempt. That same year, Congress established the National 
Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. When the 
commission published its findings five years later in the “Belmont Report,” it became the professional 
standard for imposing ethical guidelines on all federally funded research with human subjects in the US.© 

The clamorous rights consciousness of the 1970s drove behavior modification from civilian life, or at 
least dimmed its star. A Federal Bureau of Prisons official recommended that program leaders avoid 
using “the term ‘behavioral modification’ but to talk about positive reward and reinforcements for the 
type of behavior we are attempting to instill.” Another said, “We’re doing what we always did... but to 
call it ‘behavior modification’ just makes things more difficult.”© Skinner’s 1976 “primer,” titled About 
Behaviorism, motivated by what he believed were public misconceptions stirred by the wave of harsh 
reaction to Beyond Freedom & Dignity, failed to capture much public attention. According to Skinner’s 
biographer, “the battle had climaxed.” The public had made Beyond Freedom & Dignity a bestseller “but 
had just as surely rejected Skinner’s argument that there were cultural matters more important than 
preserving and extending individual freedom.” 

Most fascinating is that throughout these years of anxiety and debate, it was impossible to imagine the 
means of behavioral modification as anything other than owned and operated by the government: a 
privileged modality of state power. A 1966 Harvard Law Review article addressed issues of electronic 
tracking, surveillance, and behavioral control, reasoning that it would “consider governmental attempts to 
change conduct, since these seem more likely than private attempts.”®= The democratic impulse of US 
society, repelled by the excesses of its intelligence agencies, their support of criminal activities 
undertaken by the Nixon administration, and the migration of behavior modification as a means of 
disciplinary control in state institutions, led to the rejection of behavioral modification as an extension of 
governmental power. 

Unknown to the senators, scholars, rights activists, litigators, and many other citizens who stood 
against the antidemocratic incursions of the behavioral engineering vision, these methods had not died. 
The project would resurface in a wholly unexpected incarnation as a creature of the market, its 
unprecedented digital capabilities, scale, and scope now flourishing under the flag of surveillance 
capitalism. During the same years that US democratic forces combined to resist behavior modification as 
a form of state power, the energies of capitalist counterinsurgency were already at work within society. 
The corporation was to enjoy the rights of personhood but be free of democratic obligations, legal 
constraints, moral calculations, and social considerations. Certainly in the US case, a weakened state in 


which elected officials depend upon corporate wealth in every election cycle has shown little appetite to 
contest behavior modification as a market project, let alone to stand for the moral imperatives of the 
autonomous individual. 

In its latest incarnation, behavioral modification comes to life as a global digital market architecture 
unfettered by geography, independent of constitutional constraints, and formally indifferent to the risks it 
poses to freedom, dignity, or the sustenance of the liberal order that Ervin’s subcommittee was determined 
to defend. This contrast is even more distressing in light of the fact that in the mid-twentieth century, the 
means of behavior modification were aimed at individuals and groups who were construed as “them”: 
military enemies, prisoners, and other captives of walled disciplinary regimes. 

Today’s means of behavioral modification are aimed unabashedly at “us.” Everyone is swept up in 
this new market dragnet, including the psychodramas of ordinary, unsuspecting fourteen-year-olds 
approaching the weekend with anxiety. Every avenue of connectivity serves to bolster private power’s 
need to seize behavior for profit. Where is the hammer of democracy now, when the threat comes from 
your phone, your digital assistant, your Facebook login? Who will stand for freedom now, when 
Facebook threatens to retreat into the shadows if we dare to be the friction that disrupts economies of 
action that have been carefully, elaborately, and expensively constructed to exploit our natural empathy, 
elude our awareness, and circumvent our prospects for self-determination? If we fail to take notice now, 
how long before we are numb to this incursion and to all the incursions? How long until we notice nothing 
at all? How long before we forget who we were before they owned us, bent over the old texts of self- 
determination in the dim light, the shawl around our shoulders, magnifying glass in hand, as if deciphering 
ancient hieroglyphs? 

Throughout these chapters we have returned to the essential questions that define the division of 
learning: Who knows? Who decides? Who decides who decides? As to who knows, we have seen the 
titanic agglomerations of knowledge about our behavior in the shadow text, from vast patterns across 
populations to the intimate detail of individual lives. These new information territories are private and 
privileged, known only to the machines, their priests, and the market participants who pay to play in these 
new spaces. Although it is obviously the case that we are excluded because the knowledge is not for us, 
these chapters have revealed a deeper structural basis for our exclusion. Now we know that surveillance 
capitalists’ ability to evade our awareness is an essential condition for knowledge production. We are 
excluded because we are friction that impedes the elaboration of the shadow text and with it surveillance 
capitalism’s knowledge dominance. 

As to who decides, this division of learning has been decided by the declarations and incursions of the 
owners of private surveillance capital as another essential condition of accumulation, enabled by the 
reluctance of the state to assert democratic oversight in this secret realm. Finally, there is the question of 
who decides who decides. So far, it is the asymmetrical power of surveillance capital unencumbered by 
law that decides who decides. 

The commodification of behavior under the conditions of surveillance capitalism pivots us toward a 
societal future in which an exclusive division of learning is protected by secrecy, indecipherability, and 
expertise. Even when knowledge derived from your behavior is fed back to you in the first text as a quid 
pro quo for participation, the parallel secret operations of the shadow text capture surplus for crafting into 
prediction products destined for other marketplaces that are about you rather than for you. These markets 
do not depend upon you except first as a source of raw material from which surplus is derived, and then 
as a target for guaranteed outcomes. We have no formal control because we are not essential to the market 
action. In this future we are exiles from our own behavior, denied access to or control over knowledge 
derived from our experience. Knowledge, authority, and power rest with surveillance capital, for which 


we are merely “human natural resources.” 


CHAPTER ELEVEN 


THE RIGHT TO THE FUTURE TENSE 


But He had planned such future for this youth: 
Surely, His duty now was to compel, 
To count on time to bring true love of truth 
And, with it, gratitude. His eagle fell. 


—W. H. AUDEN 
SONNETS FROM CHINA, IX 


I. I Will to Will 


I wake early. The day begins before I open my eyes. My mind is in motion. Words and sentences have 
streamed through my dreams, solving problems on yesterday’s pages. The first work of the day is to 
retrieve those words that lay open a puzzle. Only then am I ready to awaken my senses. I try to discern 
each birdcall in the symphony outside our windows: the phoebe, redwing, blue jay, mockingbird, 
woodpecker, finch, starling, and chickadee. Soaring above all their songs are the cries of geese over the 
lake. I splash warm water on my face, drink cool water to coax my body into alertness, and commune with 
our dog in the still-silent house. I make coffee and bring it into my study, where I settle into my desk chair, 
call up my screen, and begin. I think. I write these words, and I imagine you reading them. I do this every 
day of every week—as I have for several years—and it is likely that I will continue to do so for one or 
two years to come. 

I watch the seasons from the windows above my desk: first green, then red and gold, then white, and 
then back to green again. When friends come to visit, they peek into my study. There are books and papers 
stacked on every surface and most of the floor. I know they feel overwhelmed at this sight, and sometimes 
I sense that they silently pity me for my obligation to this work and how it circumscribes my days. I do not 
think that they realize how free I am. In fact, I have never felt more free. How is this possible? 

I made a promise to complete this work. This promise is my flag planted in the future tense. It 
represents my commitment to construct a future that cannot come into being should I abandon my promise. 
This future will not exist without my capacity first to imagine its facts and then to will them into being. I 
am an inchworm moving with determination and purpose across the distance between now and later. Each 
tiny increment of territory that I traverse is annexed to the known world, as my effort transforms 
uncertainty into fact. Should I renege on my promise, the world would not collapse. My publisher would 


survive the abrogation of our contract. You would find many other books to read. I would move on to 
other projects. 

My promise, though, is an anchor that girds me against the vagaries of my moods and temptations. It is 
the product of my will to will and a compass that steers my course toward a desired future that is not yet 
real. Events may originate in energy sources outside my will and abruptly alter my course in ways that I 
can neither predict nor control. Indeed, they have already done so. Despite this certain knowledge of 
uncertainty, I have no doubt that I am free. I can promise to create a future, and I can keep my promise. If 
the book that I have imagined is to exist in the future, it must be because I will to will it so. I live in an 
expansive landscape that already includes a future that only I can imagine and intend. In my world, this 
book I write already exists. In fulfilling my promise, I make it manifest. This act of will is my claim to the 
future tense. 

To make a promise is to predict the future; to fulfill a promise through the exercise of will turns that 
prediction into fact. Our hearts pump our blood, our kidneys filter that blood, and our wills create the 
future in the patient discovery of each new sentence or step. This is how we claim our right to speak in the 
first person as the author of our futures. The philosopher Hannah Arendt devoted an entire volume to an 
examination of will as the “organ for the future” in the same way that memory is our mental organ for the 
past. The power of will lies in its unique ability to deal with things, “visibles and invisibles, that have 
never existed at all. Just as the past always presents itself to the mind in the guise of certainty, the future’s 
main characteristic is its basic uncertainty, no matter how high a degree of probability prediction may 
attain.” When we refer to the past, we see only objects, but the view to the future brings “projects,” things 
that are yet to be. With freedom of will we undertake action that is entirely contingent on our 
determination to see our project through. These are acts that we could have “left undone” but for our 
commitment. “A will that is not free,” Arendt concludes, “is a contradiction in terms.” 

Will is the organ with which we summon our futures into existence. Arendt’s metaphor of will as the 
“mental organ of our future” suggests that it is something built into us: organic, intrinsic, inalienable. 
Moral philosophers have called this “free will” because it is the human counterpoint to the fear of 
uncertainty that suffocates original action. Arendt describes promises as “islands of predictability’ and 
“suideposts of reliability’ in an “ocean of uncertainty.” They are, she argues, the only alternative to a 
different kind of “mastery” that relies on “domination of one’s self and rule over others.”* 

Centuries of debate have been levied on the notion of free will, but too often their effect has been to 
silence our own declarations of will, as if we are embarrassed to assert this most fundamental human fact. 
I recognize my direct experience of freedom as an inviolate truth that cannot be reduced to the 
behaviorists’ formulations of life as necessarily accidental and random, shaped by external stimuli 
beyond my knowledge or influence and haunted by irrational and untrustworthy mental processes that I 
can neither discern nor avoid.? 

The American philosopher John Searle, whose work on the “declaration” we discussed in Chapter 6, 
comes to a similar conclusion in his examination of “free will.” He points to the “causal gap” between the 
reasons for our actions and their enactment. We may have good reasons to do something, he observes, but 
that does not necessarily mean it will be done. “The traditional name of this gap in philosophy is ‘the 
freedom of the will.’” In response to the “sordid history” of this concept, he reasons, “even if the gap is 
an illusion it is one we cannot shake off.... The notion of making and keeping promises presupposes the 
gap.... [It] requires consciousness and a sense of freedom on the part of the promise-making and promise- 
keeping agent.”4 

The freedom of will is the existential bone structure that carries the moral flesh of every promise, and 
my insistence on its integrity is not an indulgence in nostalgia or a random privileging of the pre-digital 


human story as somehow more truly human. This is the only kind of freedom that we can guarantee 
ourselves, no matter the weight of entropy or inertia, and irrespective of the forces and fears that attempt 
to collapse time into an eternity of shadowboxing now, and now, and now. These bones are the necessary 
condition for the possibility of civilization as a “moral milieu” that favors the dignity of the individual 
and respects the distinctly human capacities for dialogue and problem solving. Any person, idea, or 
practice that breaks these bones and tears this flesh robs us of a self-authored and we-authored future. 

These principles are not quaint accessories, as Hal Varian and others suggest. Rather, they are hard- 
won achievements that have crystallized over millennia of human contest and sacrifice. Our freedom 
flourishes only as we steadily will ourselves to close the gap between making promises and keeping 
them. Implicit in this action is an assertion that through my will I can influence the future. It does not imply 
total authority over the future, of course, only over my piece of it. In this way, the assertion of freedom of 
will also asserts the right to the future tense as a condition of a fully human life. 

Why should an experience as elemental as this claim on the future tense be cast as a human right? The 
short answer is that it is only necessary now because it is imperiled. Searle argues that such elemental 
“features of human life” rights are crystallized as formal human rights only at that moment in history when 
they come under systematic threat. So, for example, the ability to speak is elemental. The concept of 
“freedom of speech” as a formal right emerged only when society evolved to a degree of political 
complexity that the freedom to speak came under threat. The philosopher observes that speech is not more 
elemental to human life than breathing or being able to move one’s body. No one has declared a “right to 
breathe” or a “right to bodily movement” because these elemental rights have not come under attack and 
therefore do not require formal protection. What counts as a basic right, Searle argues, is both 
“historically contingent” and “pragmatic.”? 

I suggest that we now face the moment in history when the elemental right to the future tense is 
endangered by a panvasive digital architecture of behavior modification owned and operated by 
surveillance capital, necessitated by its economic imperatives, and driven by its laws of motion, all for 
the sake of its guaranteed outcomes. 


IT. We Will to Will 


Most simply put, there is no freedom without uncertainty; it is the medium in which human will is 
expressed in promises. Of course, we do not only make promises to ourselves; we also make promises to 
one another. When we join our wills and our promises, we create the possibility of collective action 
toward a shared future, linked in determination to make our vision real in the world. This is the origin of 
the institution we call “contract,” beginning with the ancient Romans.® 

Contracts originated as shared “islands of predictability” intended to mitigate uncertainty for the 
human community, and they still retain this meaning. “The simplest way to state the point of contract law 
is that it supports and shapes the social practice of making and keeping promises and agreements,” 
concludes one eminent scholar. “Contract law focuses on problems of cooperation,” summarizes another. 
“Contract law... reflects a moral ideal of equal respect for persons. This fact explains why contract law 
can produce genuine legal obligations and is not just a system of coercion,” observes a third.Z 

It is in this context that the destructiveness of the uncontract is most clearly revealed. Recall Varian’s 
assertion that if someone stops making monthly car payments, “Nowadays it’s a lot easier just to instruct 
the vehicular monitoring system not to allow the car to be started and to signal the location where it can 
be picked up.” Varian calls this new capability a “new contract form,” when in reality the uncontract 


abandons the human world of legally binding promises and substitutes instead the positivist calculations 
of automated machine processes.® Without so much as a tip of the cap and a “fare-thee-well,” Varian’s 
uncontract disposes of several millennia of societal evolution during which Western civilization 
institutionalized the contract as a grand achievement of shared will. 

It is no secret that the institution of the contract has been twisted and abused in every age, from the 
Requirimiento to the “slave contract,” as incumbent power imposes painful inequalities that drain the 
meaning, and indeed the very possibility, of mutual promising.2 For example, Max Weber warned that the 
great achievements of contractual freedom create opportunities to exploit property ownership as a means 
to “the achievement of power over others.” 12 

However, today’s uncontracts are unprecedented in their ability to impose unilateral power. They 
leverage the apparatus to combine pervasive monitoring and remote actuation for an internet-enabled 
“new economics” that bypasses human promises and social engagement.t! The uncontract aims instead for 
a condition that the economist Oliver Williamson describes as “contract utopia”: a state of perfect 
information known to perfectly rational people who always perform exactly as promised./4 The problem 
is, as Williamson writes, “All complex contracts are unavoidably incomplete... parties will be 
confronted with the need to adapt to unanticipated disturbances that arise by reason of gaps, errors and 
omissions in the original contract.” 8 

If you have ever seen a house built according to architectural plans, then you have a good idea of what 
Williamson means. There is no blueprint that sufficiently details everything needed to convert drawings 
and specifications into an actual house. No plan anticipates every problem that might arise, and most do 
not come close. The builders’ skills are a function of how they collaborate to invent the actions that fulfill 
the intention of the drawings as they solve the unexpected but inevitable complications that arise along the 
way. They work together to construct a reality from the uncertainty of the plan. 

Like builders, people in contractual agreements undertake this kind of collaboration. It’s not simply 
finding the way through a maze to an already agreed-upon end point, but rather the continuous refinement 
and clarification of ends and means in the face of unanticipated obstacles. This sociality of contract may 
entail conflict, frustration, oppression, or anger, but it can also produce trust, cooperation, cohesion, and 
adaptation as the means through which human beings confront an unknowable future. 

Were “contract utopia” to exist, Williamson says, it would best be described as a “plan” that, like 
other “utopian modes,” requires “deep commitment to collective purposes” and “personal subordination.” 
Subordination to what? To the plan. Contract in this context of perfect rationality is what Williamson 
describes as “a world of planning.” Such planning was the basic institution of socialist economics, where 
the “new man” was idealized as possessing “a high level of cognitive competence” and therefore, it was 
espoused, could design highly effective plans.4 Varian deftly swaps out socialism’s “new man” and 
installs instead a market defined by surveillance capitalism’s economic imperatives, expressed through a 
ubiquitous computational architecture, the machine intelligence capabilities to which data are 
continuously supplied, the analytics that discern patterns, and the algorithms that convert them into rules. 
This is the essence of the uncontract, which transforms the human, legal, and economic risks of contracts 
into plans constructed, monitored, and maintained by private firms for the sake of guaranteed outcomes: 
less contract utopia than uncontract dystopia. 

In November 2016 the experience of three people in the small Illinois town of Belleville was 
testimony to what we lose when we subordinate ourselves to the dystopian rule of the uncontract. Pat and 
Stanford Kipping owed their credit union $350 on their 1998 Buick. Once again, they could not make their 
monthly $95 payment. The Kippings’ credit union enlisted a local repo man, Jim Ford, to take away their 
car. 


When Ford visited the Kippings’ Belleville home, he was disturbed to find an elderly couple who 
were forced to choose between buying medicine and making their car payments. Ford’s initial response 
was to waive his repossession fees. The Kippings generously thanked him, invited him in for tea, and 
shared their story. That’s when Ford decided to bridge the gap between uncertain reality and the 
stipulations of their contract. He did the human thing, calling the credit union and offering to pay the 
couple’s debt. 

The credit union manager insisted that Ford had to follow the “process.” Ford continued to invoke the 
ancient social principles of the contract, seeking a way through the maze to something that felt like justice. 
Ultimately the manager agreed to “work with” the couple to see what could be done. It didn’t end there. 
Within twenty-four hours, an online fund-raising appeal produced enough to pay off the Kippings’ car, 
detail it, purchase a Thanksgiving turkey, and give the couple an additional gift of $1,000. 

Most interesting is that when a local paper picked up the story, it quickly went viral across the web 
and traditional media. Millions of people read and responded to this drama, presumably because it stirred 
memories of something precious and necessary but now threatened with extinction. Jim Ford reminded us 
of the most cherished requirements of a civilized life: our shared assertion of rights to the future tense and 
its expression in the joining of wills in mutual commitment to dialogue, problem solving, and empathy. He 
was eloquent on this point: “Just be nice to people. It’s not that hard. The fact that this has gone so crazy is 
kind of sad. This should be a daily thing, a normal thing.”22 

In the dystopia of the uncontract, this daily human thing is not normal. What if the Kippings’ credit 
union employed Spireon’s telematics and merely had to instruct the vehicular monitoring system to 
disable the car? There would be no loan manager engaging in a give-and-take with customers. The 
algorithm tasked to eliminate the messy, unpredictable, untrustworthy eruptions of human will would have 
seized the old Buick. There would have been no shared tea time with the Kippings and no one to listen to 
their story. There would have been no opportunity to find an alternate route through the maze, no 
opportunity to build trust, no occasion for collective action, no heartwarming holiday story of kindness, 
no glimmer of hope for a human future in which the best of our institutions is preserved and fortified, no 
shared challenge of uncertainty, and no shared freedom. 

In the dystopia of the uncontract, surveillance capitalism’s drive toward certainty fills the space once 
occupied by all the human work of building and replenishing social trust, which is now reinterpreted as 
unnecessary friction in the march toward guaranteed outcomes. The deletion of uncertainty is celebrated 
as a victory over human nature: our cunning and our opportunism. All that’s left to matter are the rules that 
translate reasons into action, the objective measures of behavior, and the degree of conformance between 
the two. Social trust eventually withers, a kind of vestigial oddity like a third nipple or wisdom teeth: 
traces of an evolutionary past that no longer appear in operational form because their context and 
therefore their purpose have vanished.1® 

The uncontract and the for-profit circuits of behavior modification in which it executes its objectives 
construe society as an acrid wasteland in which mistrust is taken for granted. By positing our lives 
together as already failed, it justifies coercive intervention for the sake of certainty. Against this 
background of the gradual normalization of the automated plan and its planners, the human response of one 
repo man bears simple witness to precisely what surveillance capitalism must extinguish. 

Human replenishment from the failures and triumphs of choosing the future in the face of uncertainty 
gives way to the blankness of perpetual compliance. The word trust lingers, but its referent in human 
experience dissolves into reminiscence, an archaic footnote to a barely remembered dream of a dream 
that has long since faded for the sake of a new dictatorship of market reasons. As the dream dies, so too 
does our sense of astonishment and protest. We grow numb, and our numbness paves the way for more 


compliance. A pathological division of learning forged by unprecedented asymmetries of knowledge and 
power fixes us in a new inequality marked by the tuners and the tuned, the herders and herded, the raw 
material and its miners, the experimenters and their unwitting subjects, those who will the future and those 
who are shunted toward others’ guaranteed outcomes. 

So let us establish our bearings. Uncertainty is not chaos but rather the necessary habitat of the present 
tense. We choose the fallibility of shared promises and problem solving over the certain tyranny imposed 
by a dominant power or plan because this is the price we pay for the freedom to will, which founds our 
right to the future tense. In the absence of this freedom, the future collapses into an infinite present of mere 
behavior, in which there can be no subjects and no projects: only objects. 

In the future that the surveillance capitalism prepares for us, my will and yours threaten the flow of 
surveillance revenues. Its aim is not to destroy us but simply to author us and to profit from that 
authorship. Such means have been imagined in the past, but only now are they feasible. Such means have 
been rejected in the past, but only now have they been allowed to root. We are ensnared without 
awareness, shorn of meaningful alternatives for withdrawal, resistance, or protection. 

The promise of the promise and the will to will run deeper than these deformities. They remind us of 
that place again where we humans heal the breach between the known and the unknowable, navigating the 
seas of uncertainty in our vessels of shared promises. In the real world of human endeavor, there is no 
perfect information and no perfect rationality. Life inclines us to take action and to make commitments 
even when the future is unknown. Anyone who has brought a child into the world or has otherwise given 
his or her heart in love knows this to be true. 

Gods know the future, but we move forward, take risks, and bind ourselves to others despite the fact 
that we can’t know everything about the present, let alone the future. This is the essence of our freedom, 
expressed as the elemental right to the future tense. With the construction and ownership of the new means 
of behavioral modification, the fate of this right conforms to a pattern that we already have identified. It is 
not extinguished, but rather it is usurped: commandeered and accumulated by surveillance capital’s 
exclusive claims on our futures. 


III. How Did They Get Away with It? 


In the course of the last ten chapters I have argued that surveillance capitalism represents an 
unprecedented logic of accumulation defined by new economic imperatives whose mechanisms and 
effects cannot be grasped with existing models and assumptions. This is not to say that the old imperatives 
—a compulsion toward profit maximization along with the intensification of the means of production, 
growth, and competition—have vanished. However, these must now operate through the novel aims and 
mechanisms of surveillance capitalism. I briefly review the new imperatives here, both as a summary of 
the ground that we have covered and as prelude to the question How did they get away with it? 

Surveillance capitalism’s new story begins with behavioral surplus discovered more or less ready- 
made in the online environment, when it was realized that the “data exhaust” clogging Google’s servers 
could be combined with its powerful analytic capabilities to produce predictions of user behavior. Those 
prediction products became the basis for a preternaturally lucrative sales process that ignited new 
markets in future behavior. 

Google’s “machine intelligence” improved as the volume of data increased, producing better 
prediction products. This dynamic established the extraction imperative, which expresses the necessity 
of economies of scale in surplus accumulation and depends upon automated systems that relentlessly 


track, hunt, and induce more behavioral surplus. Google imposed the logic of conquest, defining human 
experience as free for the taking, available to be rendered as data and claimed as surveillance assets. The 
company learned to employ a range of rhetorical, political, and technological strategies to obfuscate these 
processes and their implications. 

The need for scale drove a relentless search for new high-volume supplies of behavioral surplus, 
producing competitive dynamics aimed at cornering these supplies of raw material and seeking lawless 
undefended spaces in which to prosecute these unexpected and poorly understood acts of dispossession. 
All the while, surveillance capitalists stealthily but steadfastly habituated us to their claims. In the 
process, our access to necessary information and services became hostage to their operations, our means 
of social participation fused with their interests. 

Lucrative prediction products depend upon behavioral surplus, and competition drove the supply 
challenges to a new level, expressed in the prediction imperative. More-powerful prediction products 
required economies of scope as well as scale, variation as well as volume. This variation occurs along 
two dimensions. The first is extension across a wide range of activities; the second is the depth of 
predictive detail within each activity. 

In this new phase of competitive intensity, surveillance capitalists are forced from the virtual world 
into the real one. This migration necessitates new machine processes for the rendition of all aspects of 
human experience into behavioral data. Competition now occurs in the context of a rapidly evolving 
global architecture of ubiquitous computation and therefore ubiquitous supply opportunities, as prediction 
products are increasingly expected to approximate certainty and therefore to guarantee behavioral 
outcomes. 

In a third phase of competitive intensity, surveillance capitalists discovered the necessity of 
economies of action based on new methods that go beyond tracking, capturing, analyzing, and predicting 
behavior in order to intervene in the state of play and actively shape behavior at the source. The result is 
that the means of production are subordinated to an elaborate new means of behavioral modification, 
which relies upon a variety of machine processes, techniques, and tactics (tuning, herding, conditioning) 
to shape individual, group, and population behavior in ways that continuously improve their 
approximation to guaranteed outcomes. Just as industrial capitalism was driven to the continuous 
intensification of the means of production, so surveillance capitalists are now locked in a cycle of 
continuous intensification of the means of behavioral modification. 

Surveillance capitalists’ interests have shifted from using automated machine processes to know about 
your behavior to using machine processes to shape your behavior according to their interests. In other 
words, this decade-and-a-half trajectory has taken us from automating information flows about you to 
automating you. Given the conditions of increasing ubiquity, it has become difficult if not impossible to 
escape this audacious, implacable web. 

In order to reestablish our bearings, I have asked for a rebirth of astonishment and outrage. Most of 
all, I have asked that we reject the Faustian pact of participation for dispossession that requires our 
submission to the means of behavioral modification built on the foundation of the Google declarations. I 
am also mindful, though, that when we ask How did they get away with it? there are many compelling 
reasons to consider, no one of which stands alone. Instead of simple cause and effect, the answers to our 
question constitute a broad landscape of history, contingency, quicksand, and coercion. 

Our question is even more vexing in light of the fact that in the great majority of surveys designed to 
probe public attitudes toward the loss of privacy and other elements of surveillance capitalist practices, 
few of us favor the status quo. In forty-six of the most prominent forty-eight surveys administered between 
2008 and 2017, substantial majorities support measures for enhanced privacy and user control over 


personal data. (Only two early surveys were somewhat less conclusive, because so many participants 
indicated that they did not understand how or what personal information was being gathered.) Indeed, by 
2008 it was well established that the more knowledge one has about “internet privacy practices,” the 
more one is likely to be very concerned about privacy.“2 

Although the surveys vary in terms of their specific focus and questions, the general consistency of 
responses over the decade is noteworthy. For example, an important 2009 survey found that when 
Americans are informed of the ways that companies gather data for targeted online ads, 73-86 percent 
reject such advertising. Another substantial survey in 2015 found 91 percent of respondents disagreeing 
that the collection of personal information “without my knowing” is a fair trade-off for a price discount. 
Fifty-five percent disagreed that it was a fair exchange for improved services. In 2016 Pew Research 
reported only 9 percent of respondents as very confident in trusting social media sites with their data and 
14 percent as very confident about trusting companies with personal data. More than 60 percent wanted to 
do more to protect their privacy and believed there should be more regulations to protect privacy./8 

Surveillance capitalist firms have tended to dismiss these survey results, pointing instead to the 
spectacular growth of users and revenue. This discrepancy has confounded research and public policy. 
With so many people rejecting the practices of surveillance capitalism, even considering how little most 
of us actually know about these practices, how is it that this market form has been able to succeed? The 
reasons are plentiful: 

1. Unprecedented: Most of us did not resist the early incursions of Google, Facebook, and other 
surveillance capitalist operations because it was impossible to recognize the ways in which they differed 
from anything that had gone before. The basic operational mechanisms and business practices were so 
new and strange, so utterly sui generis, that all we could see was a gaggle of “innovative” horseless 
carriages. Most significantly, anxiety and vigilance have been fixed on the known threats of surveillance 
and control associated with state power. Earlier incursions of behavior modification at scale were 
understood as an extension of the state, and we were not prepared for the onslaught from private firms. 

2. Declaration as invasion: The lack of precedence left us disarmed and charmed. Meanwhile, 
Google learned the art of invasion by declaration, taking what it wanted and calling it theirs. The 
corporation asserted its rights to bypass our awareness, to take our experience and transform it into data, 
to claim ownership of and decisions over the uses of those data, to produce strategies and tactics that 
keep us ignorant of its practices, and to insist on the conditions of lawlessness required for these 
operations. These declarations institutionalized surveillance capitalism as a market form. 

3. Historical context: Surveillance capitalism found shelter in the neoliberal zeitgeist that equated 
government regulation of business with tyranny. This “paranoid style” favored self-management regimes 
that imposed few limits on corporate practices. In a parallel development, the “war on terror” shifted the 
government’s attention from privacy legislation to an urgent interest in the rapidly developing skills and 
technologies of Google and other rising surveillance capitalists. These “elective affinities” produced a 
trend toward surveillance exceptionalism, which further sheltered the new market form from scrutiny and 
nurtured its development. 

4. Fortifications: Google aggressively protected its operations by establishing its utilities in the 
electoral process, strong relationships with elected and appointed officials, a revolving door of staffers 
between Washington and Silicon Valley, lavish lobbying expenditures, and a steady “soft-power” 
campaign of cultural influence and capture. 

5. The dispossession cycle: First at Google and later at Facebook and other firms, surveillance 
capitalist leaders mastered the rhythms and stages of dispossession. Audacious incursions are pursued 
until resistance is met, followed by a range of tactics from elaborate public relations gambits to legal 


combat, all designed to buy time for gradual habituation to once-outrageous facts. A third stage features 
public demonstrations of adaptability and even retreat, while in the final stage resources are redirected to 
achieve the same objectives camouflaged by new rhetoric and tactics. 

6. Dependency: The free services of Google, Facebook, and others appealed to the latent needs of 
second-modernity individuals seeking resources for effective life in an increasingly hostile institutional 
environment. Once bitten, the apple was irresistible. As surveillance capitalism spread across the 
internet, the means of social participation become coextensive with the means of behavioral modification. 
The exploitation of second-modernity needs that enabled surveillance capitalism from the start eventually 
imbued nearly every channel of social participation. Most people find it difficult to withdraw from these 
utilities, and many ponder if it is even possible. 

7. Self-Interest: New markets in future behavior give rise to networks of fellow travelers, partners, 
collaborators, and customers whose revenues depend on the prediction imperative. Institutional facts 
proliferate. The pizzeria owner on the Pokémon Go map, the merchant who saturates his shop with 
beacons, and the insurance companies vying for behavioral data unite in the race toward guaranteed 
outcomes and surveillance revenues. 

8. Inclusion: Many people feel that if you are not on Facebook, you do not exist. People all over the 
world raced to participate in Pokémon Go. With so much energy, success, and capital flowing into the 
surveillance capitalist domain, standing outside of it, let alone against it, can feel like a lonely and risky 
prospect. 

9. Identification: Surveillance capitalists aggressively present themselves as heroic entrepreneurs. 
Many people identify with and admire the financial success and popularity of the surveillance capitalists 
and regard them as role models. 

10. Authority: Many also regard these corporations and their leaders as authorities on the future: 
geniuses who can see farther than the rest of us. It is easy to fall prey to the naturalistic fallacy, which 
suggests that because the companies are successful, they must also be right. As a result, many of us are 
respectful of these leaders’ expert status and are eager to participate in innovations that anticipate the 
future. 

11. Social persuasion: As we have seen repeatedly, there is an endless cascade of beguiling rhetoric 
aimed at persuading people of the wonders associated with surveillance capitalist innovations: targeted 
advertising, personalization, and digital assistants. Beyond that, economies of action are intentionally 
designed to persuade us to follow one another along prescribed courses of action. 

12. Foreclosed alternatives: The “dictatorship of no alternatives” is in full force here. We have seen 
that the behavioral value reinvestment cycle is increasingly rare. The Aware Home gave way to the 
Google Home. Surveillance capitalism spread across the internet, and the drive toward economies of 
scope and action has forced it out into the real world. From apps to devices to the One Voice, it is ever 
more difficult to identify avenues of escape, let alone genuine alternatives. 

13. Inevitabilism: The Trojan horse of computer mediation—devices, apps, connection—enters the 
scene ina relentless deluge of inevitabilist rhetoric, successfully distracting us from the highly intentional 
and historically contingent surveillance capitalism within. New institutional facts proliferate and stabilize 
the new practices. We fall into resignation and a sense of helplessness. 

14. The ideology of human frailty: In addition to inevitabilism, surveillance capitalism has eagerly 
weaponized behavioral economics’ ideology of human frailty, a worldview that frames human mentation 
as woefully irrational and incapable of noticing the regularity of its own failures. Surveillance capitalists 
employ this ideology to legitimate their means of behavior modification: tuning, herding, and conditioning 
individuals and populations in ways that are designed to elude awareness. 


15. Ignorance: This remains a salient explanation. Surveillance capitalists dominate an abnormal 
division of learning in which they know things that we cannot know while compelled to conceal their 
intentions and practices in secret backstage action. It is impossible to understand something that has been 
crafted in secrecy and designed as fundamentally illegible. These systems are intended to ensnare us, 
preying on our vulnerabilities bred by an asymmetrical division of learning and amplified by our scarcity 
of time, resources, and support. 

16. Velocity: Surveillance capitalism rose from invention to domination in record time. This reflects 
its ability to attract capital and its laws of motion, but it also reflects a specific strategy in which velocity 
is consciously deployed to paralyze awareness and freeze resistance while distracting us with immediate 
gratifications. Surveillance capitalism’s velocities outrun democracy even as they outrun our ability to 
understand what is happening and consider the consequences. This strategy is borrowed from a long 
legacy of political and military approaches to the production of speed as a form of violence, most recently 
known as “shock and awe.”!2 

These sixteen answers suggest that in the nearly two decades since the invention of surveillance 
capitalism, existing laws, largely centered on privacy and antitrust, have not been sufficient to disrupt its 
growth. We need laws that reject the fundamental legitimacy of surveillance capitalism’s declarations and 
interrupt its most basic operations, including the illegitimate rendition of human experience as behavioral 
data; the use of behavioral surplus as free raw material; extreme concentrations of the new means of 
production; the manufacture of prediction products; trading in behavioral futures; the use of prediction 
products for third-order operations of modification, influence, and control; the operations of the means of 
behavioral modification; the accumulation of private exclusive concentrations of knowledge (the shadow 
text); and the power that such concentrations confer. 

The rejection of these new institutions of surveillance capital and the declarations upon which they are 
built would signify a withdrawal of social agreement to surveillance capitalism’s aims and methods in the 
Same way that we once withdrew agreement to the antisocial and antidemocratic practices of raw 
industrial capitalism, righting the balance of power between employers and workers by recognizing 
workers’ rights to collective bargaining and outlawing child labor, hazardous working conditions, 
excessive work hours, and so on. 

The withdrawal of agreement takes two broad forms, a distinction that will be useful as we move into 
Part III. The first is what I call the counter-declaration. These are defensive measures such as encryption 
and other privacy tools, or arguments for “data ownership.” Such measures may be effective in discrete 
situations, but they leave the opposing facts intact, acknowledging their persistence and thus paradoxically 
contributing to their legitimacy. For example, if I “opt out” of tracking, I opt out for me, but my action 
does not challenge or alter the offending practice. The second form of disagreement is what I call the 
synthetic declaration. If the declaration is “check,” the counter-declaration is “checkmate,” and the 
synthetic declaration changes the game. It asserts an alternative framework that transforms the opposing 
facts. We bide our time with counter-declarations and make life more tolerable, but only a synthetic 
alternative vision will transform raw surveillance capitalism in favor of a digital future that we can call 
home. 

I turn to the history of the Berlin Wall as an illustration of these two forms of disagreement. From 1961 
through the early 1980s, courageous East Berliners carved seventy-one tunnels through the sandy soil 
beneath the city, affording several hundred people a means of escape to West Berlin.“ The tunnels are 
testament to the necessity of counter-declarations, but they did not bring down the wall or the power that 
sustained it. 

The synthetic declaration gathered force over decades, but its full expression would have to wait until 


near midnight on November 9, 1989, when Harald Jager, the senior officer on duty that night at the 
Bornholmer Street passage, gave the order to open the gates, and twenty thousand people surged across 
the wall into West Berlin. As one historian describes that event, “By the night of November 9, when the 
people appeared at the Berlin Wall and demanded to know of the border officials, Will you let us pass?, 
those people had become so certain of themselves, and the officials so unsure of themselves, that the 
answer was We will.” 


IV. Prophecy 


Nearly seventy years ago, the economic historian Karl Polanyi reflected on the ways in which industrial 
capitalism’s market dynamics would, if left unchecked, destroy the very things that it aimed to buy and 
sell: “The commodity fiction disregarded the fact that leaving the fate of soil and people to the market 
would be tantamount to annihilating them.”22 In the absence of synthetic declaration, Polanyi’s prophecy 
appears headed for fulfillment, and this fact alone must put us on alert. What does Polanyi’s prophecy 
augur for our time? 

Industrial capitalism followed its own logic of shock and awe, taking aim at nature to conquer “it” in 
the interests of capital; now surveillance capitalism has human nature in its sights. We have only 
gradually come to understand that the specific methods of domination employed by industrial capitalism 
for more than two centuries have fundamentally disoriented the conditions that support life on Earth, 
violating the most basic precepts of civilization. Despite the many benefits and immense accomplishments 
of industrial capitalism, it has left us perilously close to repeating the fate of the Easter Islanders, who 
wrecked the ground that gave them life, then fashioned statues to scan the horizon for the aid and succor 
that would never come. If industrial capitalism dangerously disrupted nature, what havoc might 
surveillance capitalism wreak on human nature ? 

The answer to this question requires a return to imperatives. Industrial capitalism brought us to the 
brink of epic peril, but not as a consequence of an evil lust for destruction or runaway technology. Rather, 
this result was ineluctably driven by its own inner logic of accumulation, with its imperatives of profit 
maximization, competition, the relentless drive for labor productivity through the technological 
elaboration of production, and growth funded by the continuous reinvestment of surplus.*? It is Weber’s 
“economic orientation” that matters, and how that orientation merges with the specific form of capitalism 
that rises to dominance in each age. 

The logic of industrial capitalism exempted the enterprise from responsibility for its destructive 
consequences, unleashing the destabilization of the climate system and the chaos it spells for all creatures. 
Polanyi understood that raw capitalism could not be cooked from within. He argued that it was up to 
society to impose those obligations on capitalism by insisting on measures that tether the capitalist project 
to the social, preserving and sustaining life and nature. 

Similarly, the meaning of Polanyi’s prophecy for us now can be grasped only through the lens of 
surveillance capitalism’s economic imperatives as they frame its claim to human experience. If we are to 
rediscover our sense of astonishment, then let it be here: if industrial civilization flourished at the 
expense of nature and now threatens to cost us the Earth, an information civilization shaped by 
surveillance capitalism will thrive at the expense of human nature and threatens to cost us our 
humanity. Polanyi’s prophecy requires us to ask if we may yet avert this fate with our own synthetic 
declarations. 

Parts I and II have been devoted to understanding the origins of surveillance capitalism and 


identifying, naming, and scrutinizing its foundational mechanisms and economic imperatives. The idea 
from the start was that naming and taming are inextricable, that fresh and careful naming can better equip 
us to intercept these mechanisms of dispossession, reverse their action, produce urgently needed friction, 
challenge the pathological division of learning, and ultimately synthesize new forms of information 
capitalism that genuinely meet our needs for effective life. Social participation and individual 
effectiveness should not require the sacrifice of our right to the future tense, which comprises our will to 
will, our autonomy, our decision rights, our privacy, and, indeed, our human natures. 

However, it would be wrong to suppose that surveillance capitalism can be grasped solely through the 
prism of its economic action or that the challenges we face are restricted to discerning, containing, and 
transforming its foundational mechanisms. The consequences of this new logic of accumulation have 
already leaked and continue to leak beyond commercial practices into the fabric of our social relations, 
transforming our relationships to ourselves and to one another. These transformations provide the soil in 
which surveillance capitalism has flourished: an invasive species that creates its own food supply. In 
transforming us, it produces nourishment for its own march forward. 

It is easier, perhaps, to see these dynamisms by looking to the past. The difference between industrial 
capitalism and industrial civilization is the difference between the economic operation and the societies it 
produced. The variant of industrial capitalism that rose to dominance in the late nineteenth and early 
twentieth centuries produced a specific kind of moral milieu that we sense intuitively even when we do 
hot name it. 

Industrial capitalism was marked by the specialized division of labor, with its historically specific 
characteristics: the conversion of craft work to mass production based on standardization, rationalization, 
and the interchangeability of parts; the moving assembly line; volume production; large populations of 
wage earners concentrated in factory settings; professionalized administrative hierarchies; managerial 
authority; functional specialization; and the distinction between white-collar work and blue-collar work. 

The list is illustrative, not exhaustive, but enough to remind us that industrial civilization was drawn 
from these expressions of the economic imperatives that ruled industrial expansion. The division of labor 
shaped culture, psychology, and social experience. The shift from craft to hourly wages created new 
populations of employees and consumers, men and women wholly dependent on the means of production 
owned and operated by private firms. 

This was the crucible of mass society, its hierarchical authority, and its centralized bureaucratic forms 
of public and private power, all of it haunted by the specters of conformity, obedience, and human 
standardization. Lives were defined by institutions that mirrored industrial organization: schools, 
hospitals, and even aspects of family and domestic life, in which ages and stages were understood as 
functions of the industrial system, from training to retirement. 

At a time when surveillance capitalism has emerged as the dominant form of information capitalism, 
we must ask the question: what kind of civilization does it foretell? The chapters that follow in Part III are 
intended as an initial contribution to this urgent conversation. I have said that there can be no commitment 
to “guaranteed outcomes” without the power to make it so. What is the nature of this new power? How 
will it transform our societies? What solution for a third modernity does it proffer? What novel struggles 
will haunt these new days, and what do they portend for a digital future that we can call home? These are 
the questions that guide us into Part III. 


CHAPTER TWELVE 


TWO SPECIES OF POWER 


So an age ended, and its last deliverer died 
In bed, grown idle and unhappy; they were safe: 
The sudden shadow of a giant’s enormous calf 
Would fall no more at dusk across their lawns outside. 


—W. H. AUDEN 
SONNETS FROM CHINA, X 


I. A Return to the Unprecedented 


Under surveillance capitalism, the “means of production” serves the “means of behavioral modification.” 
Machine processes replace human relationships so that certainty can replace trust. This new assembly 
relies upon a vast digital apparatus, world-historic concentrations of advanced computational knowledge 
and skill, and immense wealth. The arc of behavioral modification at scale integrates the many operations 
that we have examined: ubiquitous extraction and rendition, actuation (tuning, herding, conditioning), 
behavioral surplus supply chains, machine-intelligence—based manufacturing processes, fabrication of 
prediction products, dynamic behavioral futures markets, and “targeting,” which leads to fresh rounds of 
tuning, herding, conditioning, and the coercions of the uncontract, thus renewing the cycle. 

This assembly is a market project: its purpose is to fabricate predictions, which become more 
valuable as they approach certainty. The best predictions feed on totalities of data, and on the strength of 
this movement toward totality, surveillance capitalists have hijacked the division of learning in society. 
They command knowledge from the decisive pinnacle of the social order, where they nourish and protect 
the shadow text: the urtext of certainty. This is the market net in which we are snared. 

In Parts I and II we examined the conditions, mechanisms, and operations that construct this private 
knowledge kingdom and its lucrative predictions that evolve toward certainty in order to guarantee market 
players the outcomes that they seek. As I wrote in Chapter 7, there can be no guarantee of outcomes 
without the power to make it so. This is the dark heart of surveillance capitalism: a new type of 
commerce that reimagines us through the lens of its own distinctive power, mediated by its means of 
behavioral modification. What is this power, and how does it remake human nature for the sake of its 
lucrative certainties? 

As to this species of power, I name it instrumentarianism, defined as the instrumentation and 


instrumentalization of behavior for the purposes of modification, prediction, monetization, and 
control. In this formulation, “instrumentation” refers to the puppet: the ubiquitous connected material 
architecture of sensate computation that renders, interprets, and actuates human experience. 
“Instrumentalization” denotes the social relations that orient the puppet masters to human experience as 
surveillance capital wields the machines to transform us into means to others’ market ends. Surveillance 
capitalism forced us to reckon with an unprecedented form of capitalism. Now the instrumentarian power 
that sustains and enlarges the surveillance capitalist project compels a second confrontation with the 
unprecedented. 

When scholars, civil society leaders, journalists, public figures, and, indeed, most of us speak out 
courageously against this new power, invariably we look to Orwell’s Big Brother and more generally the 
specter of totalitarianism as the lens through which to interpret today’s threats. Google, Facebook, and the 
larger field of commercial surveillance are frequently depicted as “digital totalitarianism.”! I admire 
those who have stood against the incursions of commercial surveillance, but I also suggest that the 
equation of instrumentarian power with totalitarianism impedes our understanding as well as our ability 
to resist, neutralize, and ultimately vanquish its potency. There is no historical precedent for 
instrumentarianism, but there is vivid precedent for this kind of encounter with an unprecedented new 
species of power. 

In the years before totalitarianism was named and formally analyzed, its critics appropriated the 
language of imperialism as the only framework at hand with which to articulate and resist the new 
power’s murderous threats. Now surveillance capitalism has cast us adrift in another odd, dark sea of 
novel and thus indiscernible dangers. As scholars and citizens did before us, it is we who now reach for 
familiar vernaculars of twentieth-century power like lifesaving driftwood. 

We are back to the syndrome of the horseless carriage, where we attach our new sense of peril to old, 
familiar facts, unaware that the conclusions to which they lead us are necessarily incorrect. Instead, we 
need to grasp the specific inner logic of a conspicuously twenty-first-century conjuring of power for 
which the past offers no adequate compass. Totalitarianism was bent on the reconstruction of the human 
species through the dual mechanisms of genocide and the “engineering of the soul.” Instrumentarian 
power, as we Shall see, takes us ina sharply different direction. Surveillance capitalists have no interest 
in murder or the reformation of our souls. Although their aims are in many ways just as ambitious as those 
of totalitarian leaders, they are also utterly distinct. The work of naming a strange form of power 
unprecedented in the human experience must begin anew for the sake of effective resistance and the 
creative power to insist on a future of our own making. 

The remainder of this chapter prepares the way. The first task is to develop our understanding of what 
instrumentarian power is not, so in the section that follows we briefly consider key elements of twentieth- 
century totalitarianism. Most important is the fact that, like instrumentarianism, totalitarian power was 
also unprecedented. It literally defied human comprehension. There is much that we can learn from the 
struggles and missteps of scholars, journalists, and citizens as they found themselves overwhelmed by a 
force that they could neither fathom nor resist. Once we have tackled these questions, we will be 
equipped to bear down on an exploration of instrumentarianism’s origins in a field of intellectual 
endeavor that came to be known as “radical behaviorism,” most notably championed by B. F. Skinner and 
his dream of a “technology of behavior.” In Chapter 13 we integrate our insights to consider the unique 
aims and strategies of instrumentarian power. 


IT. Totalitarianism as a New Species of Power 


The word totalitarianism first appeared in the early-twentieth-century work of Italian philosopher 
Giovanni Gentile and came into wider use later with Mussolini’s The Doctrine of Fascism, cowritten in 
1932 with Gentile, who was by then Italy’s premier philosopher of fascism.’ Italy had entered the 
twentieth century as a second-rate country, ignored on the world stage, nursing a sense of failure and 
humiliation, and unable to support its own population as millions emigrated in search of a better life. In 
the first decade of the twentieth century a new generation of intellectuals and avant-garde futurists began 
to weave the dream of a “new Italy.” Gentile dedicated his philosophical talents to this revival of 
nationalist zeal. 

At the heart of Gentile’s political philosophy is the concept of the “total.”2 The state was to be 
understood as an inclusive organic unity that transcends individual lives. All separateness and difference 
are surrendered to the state for the sake of this superordinate totality. In 1932 Mussolini charged Gentile 
with writing the philosophical introduction to his book, while Mussolini authored the social and political 
principles that would define the fascist worldview.4 The Doctrine begins by declaring the fascist attitude 
as, above all, “a spiritual attitude” that penetrates the most intimate redoubt of each human participant: 


To know men one must know man.... Fascism is totalitarian and the Fascist State—a synthesis and a unit inclusive of all values— 
interprets, develops, and potentates the whole life of a people.... [It] is an inwardly accepted standard and rule of conduct, a 
discipline of the whole person; it permeates the will no less than the intellect... sinking deep down into his personality; it dwells in the 


heart of the man of action and of the thinker, of the artist, and of the man of science: soul of the soul.... It aims at refashioning not 


only the forms of life but their content—man, his character, and his faith... entering into the soul and ruling with undisputed sway2 


That year, the refashioning of the soul as the hallmark of the totalitarian impulse was immortalized by 
Stalin on a glittering, champagne-soaked Moscow evening. The setting was an auspicious literary 
gathering hosted by a compliant Maxim Gorky in the sprawling mansion that Stalin had presented to the 
revered author upon his return to Russia from a self-imposed Italian exile. Stalin took the floor for a toast 
as the room fell silent. “Our tanks are worthless if the souls who must steer them are made of clay. This is 
why I say: The production of souls is more important than that of tanks.... Man is reshaped by life itself, 
and those of you here must assist in reshaping his soul. That is what is important, the production of human 
souls. And that is why I raise my glass to you, writers, to the engineers of the soul.”® The authors 
assembled around Stalin that evening raised their glasses to his toast, persuaded perhaps by memories of 
less adaptive colleagues already exiled or executed, including the 1929 torture and murder of artists and 
writers in the Solovetsky Islands’ aptly named Church of the Beheading.2 

By 1933, the term totalitarianism had begun to circulate widely in Germany. Minister of Propaganda 
Joseph Goebbels adopted it, and German intellectuals declared the “turn to totalitarianism.” Nazism also 
shifted the doctrine in an important way, asserting the “movement,” not the “state,” as the spiritual center 
of German totalitarianism, a relationship summarized during Hitler’s first years as chancellor in National 
Socialism’s popular slogan “The movement gives orders to the state.”8 

That totalitarianism was a new species of power had confounded its analysis from the start, as both its 
Russian and German variants swept through those societies, challenging the foundations of Western 
civilization. Although these totalitarian regimes began to take root years before World War II—first in 
Russia in 1929 with Stalin’s ascension to power and then in Germany in 1933 with Hitler’s installation as 
chancellor—they eluded systematic study until the end of the war. Analysis was impeded in part by the 
sheer mystery and perpetual movement of the whole enterprise: the secret plans executed by secret police, 
the silent complicities and hidden atrocities, the ceaseless transformation of who or what was up or 
down, the intentional torsion of facts into anti-facts accompanied by a perpetual deluge of propaganda, 


misinformation, euphemism, and mendacity. The authoritative leader, or “egocrat,” to use the French 
philosopher Claude Lefort’s term, displaces the rule of law and “common” sense to become the quixotic 
judge of what is just or unjust, truth or lie, at each moment.2 

Western publics, especially in the US, were genuinely unable to grasp the enormity of what was 
underway. It literally boggled minds. This intellectual paralysis is immortalized in the pages of a cultural 
icon of that era, Look magazine. Its August 15, 1939, issue featured an article titled “What’s Going On in 
Russia?” written by New York Times former Moscow bureau chief and Pulitzer Prize winner Walter 
Duranty./2 The piece appeared just months after the conclusion of the Great Terror, when, between 1937 
and 1938, Stalin ordered the murders of whole sectors of the Soviet population, from poets to diplomats, 
generals to political loyalists. According to Soviet historian Robert Conquest, that two-year period saw 
seven million arrests, one million executions, two million deaths in labor camps, one million people 
imprisoned, and another seven million people still in camps by the end of 1938.4 

Despite the immediacy of catastrophic evil, Duranty’s article describes the Constitution of the USSR 
as one of the “the most democratic in the world... a foundation on which a future democracy may be 
built.” In addition to praiseworthy descriptions of the Red Army, free education and medical care, 
communal housing, and equality of the sexes, there is an upbeat commentary in which the “great purge” is 
breezily described as “one of the periodic cleansings of the Communist Party.” Duranty reports that this 
“cleansing” is “now over,” and people are “repairing the damage,” as if the country were tidying up after 
a particularly nasty winter storm. In fact, the Stalinist pattern of violence, imprisonment, exile, and 
execution merely shifted its focus with a swift and terrifying ferocity to the Baltic and eastern Poland. 
Among the many atrocities between 1939 and 1941, hundreds of thousands of Poles were marched to 
northern labor camps,/* and tens of thousands of members of the Polish Communist Party were 
murdered./2 Just one week after Duranty’s article, Stalin signed a nonaggression pact with Hitler, attacked 
Poland in September, and in November the Red Army invaded Finland.“ In 1940 Stalin ordered the 
massacre of 15,000 Polish nationalists taken as prisoners of war in the 1939 attack.12 

The most startling feature of Duranty’s essay is its characterization of Stalin himself. There, 
sandwiched between a celebratory feature on a newly released film called The Wizard of Oz and a 
lengthy spread on embarrassing celebrity pictures, such as the famed ventriloquist’s dummy Charlie 
McCarthy with a cigarette in his wooden mouth, is a photo of a handsome smiling Joseph Stalin captioned 
“Stalin, chairman of the inner circle of the Communist Party... does not lay down the law as Lenin did. 
Stalin prefers to hear the views of his associates before he makes his own decision.”® Stalin’s 1939 
lionization in Look as an exemplar of participatory management was followed a few months later with his 
ascension to Time magazine’s cover as “Man of the Year.” Indeed, between 1930 and his death in 1953, 
Stalin made a total of ten appearances on the cover of Time. All of this provides some sense of 
totalitarianism’s elaboration and institutionalization long before it was identified and analyzed as a 
coherent new form of power that, as many scholars would conclude, posed history’s greatest threat to 
civilization.“ 

With a few important exceptions, it was only after the Nazi defeat that the program of naming began in 
earnest. “Plenty of information was available contradicting the official picture,” writes Conquest. He asks 
why “journalists, sociologists, and other visitors” were taken in by the Soviet regime’s lies. One reason is 
that the Soviet government went to a great deal of trouble to present a false picture, including “model 
prisons” that betrayed no trace of the immense state machinery of torture and death. Another reason was 
the credulity of the observers themselves. In some cases, like Duranty’s, they were blinded by an 
ideological allegiance to the idea of a socialist state.18 

The most compelling reason of all is that in most of these cases, journalists, scholars, and Western 


governments had a difficult time reckoning the full weight of totalitarianism’s monstrous achievements 
because the actual facts were so “improbable” that it was difficult even for specialists to grasp their truth. 
“The Stalin epoch,” writes Conquest, “is replete with what appear as improbabilities to minds unfitted to 
deal with the phenomena.”!? This failure of comprehension has immediate significance for us as we learn 
to reckon with surveillance capitalism and its new instrumentarian power. 

The confrontation with totalitarianism’s impossibility is reflected in the moving accounts of the first 
scholars determined to lift the veil on that era’s gruesome truths. Nearly every intellectual who turned to 
this project in the period immediately following the war cites the feeling of astonishment at the 
suddenness with which, as Harvard political scientist Carl Friedrich put it, totalitarianism had “burst 
upon mankind... unexpected and unannounced.”2 Its manifestations were so novel and unanticipated, so 
shocking, rapid, and unparalleled, that all of it eluded language, challenging every tradition, norm, value, 
and legitimate form of action. The systematic accretion of violence and complicity that engulfed whole 
populations at extreme velocity invoked a kind of bewilderment that ended in paralysis, even for many of 
the greatest minds of the twentieth century. 

Friedrich was among the first scholars of totalitarianism to address this experience of improbability, 
writing in 1954 that “virtually no one before 1914 anticipated the course of development which has 
overtaken Western civilization since then... none of the outstanding scholars in history, law, and the social 
sciences discerned what was ahead... which culminated in totalitarianism. To this failure to foresee 
corresponds a difficulty in comprehending.”*! Not even the most farsighted of the early-century 
interpreters of industrial society, thinkers such as Durkheim and Weber, had anticipated this murderous 
turn. Hannah Arendt described the defeat of Nazi Germany as “the first chance to try to tell and to 
understand what had happened... still in grief and sorrow and... a tendency to lament, but no longer in 
speechless outrage and impotent horror.” 

Ultimately, a courageous and brilliant body of scholarship would evolve to meet the challenge of 
comprehension. It yielded different models and schools of thought, each with distinct emphasis and 
insights, but these shared common purpose in finally naming the great evil. “Totalitarianism has 
discovered a means of dominating and terrorizing human beings from within,” wrote Arendt, the German- 
born philosopher who would spend the six years after World War II writing her extraordinary study of 
totalitarian power, published in 1951 as The Origins of Totalitarianism.” 

Arendt’s was a detailed disclosure and a pioneering attempt to theorize what had just occurred. 
“Comprehension,” she said, is the necessary response to the “truly radical nature of Evil” disclosed by 
totalitarianism. “It means... examining and bearing consciously the burden which our century has placed 
on us—neither denying its existence nor submitting meekly to its weight.” Totalitarianism was bent on the 
“destruction of humanity’ and “the essence of man,” and, she insisted, “to turn our backs on the 
destructive forces of the century is of little avail.”“4 Essential to totalitarianism was the deletion of all 
ties and sources of meaning other than “the movement”: “Total loyalty—the psychological basis for 
domination—can be expected only from the completely isolated human being who, without any other 
social ties to family, friends, comrades, or even mere acquaintances, derives his sense of having a place 
in the world only from his belonging to a movement, his membership in the party.” 

Mid-century scholars such as Friedrich, Adorno, Gurian, Brzezinski, and Aron added to these themes, 
recognizing totalitarianism’s insistence on domination of the human soul.22 To command populations right 
down to their souls requires unimaginable effort, which was one reason why totalitarianism was 
unimaginable. It requires henchmen, and their henchmen, and their henchmen, all willing to roll up both 
sleeves and thrust both hands into the blood and shit of actual living persons whose bodies stink and 
sweat and cry out in terror, grief, and pain.2’ It measures success at the cellular level, penetrating to the 


quick, where it subverts and commands each unspoken yearning in pursuit of the genocidal vision that 
historian Richard Shorten calls “the experiment in reshaping humanity.”2° 

The destruction and reconstruction of society and the purification of the human species were 
prosecuted in the name of “class” in Stalin’s Soviet Union and “race” in Hitler’s Germany. Each regime 
invented “out-groups” slated for murder—the Jewish people, Romanies, homosexuals, and 
revolutionaries in Germany and Eastern Europe, whole segments of the population in Stalin’s Russia— 
and “in-groups” required to submit body and soul to the regime.*2 In this way totalitarian regimes could 
achieve their fantastical aim of the “People-as-one,” as Claude Lefort describes it: “Social unanimity 
corresponds to inner unanimity, held in place by hatred activated toward the ‘enemies of the people.’”2 

Totalitarian power cannot succeed by remote control. Mere conformity is insufficient. Each individual 
inner life must be claimed and transformed by the perpetual threat of punishment without crime. Mass 
murder warrants economies of scale—the camps, massacres, and gulags—but for the rest it would be a 
handmade terror that aims to remake every aspect of the individual from the inside out: heart, mind, 
sexuality, personality, and spirit. This craftwork requires the detailed orchestration of isolation, anxiety, 
fear, persuasion, fantasy, longing, inspiration, torture, dread, and surveillance. Arendt describes the 
relentless process of “atomization” and fusion in which terror destroys the ordinary human bonds of law, 
norms, trust, and affection, “which provide the living space for the freedom of the individual.” The “iron 
band” of terror “mercilessly presses men... against each other so that the very space of free action... 
disappears.” Terror “fabricates the oneness of all men.”2! 


III, An Opposite Horizon 


Instrumentarian power moves differently and toward an opposite horizon. Totalitarianism operated 
through the means of violence, but instrumentarian power operates through the means of behavioral 
modification, and this is where our focus must shift. Instrumentarian power has no interest in our souls or 
any principle to instruct. There is no training or transformation for spiritual salvation, no ideology against 
which to judge our actions. It does not demand possession of each person from the inside out. It has no 
interest in exterminating or disfiguring our bodies and minds in the name of pure devotion. It welcomes 
data on the behavior of our blood and shit, but it has no interest in soiling itself with our excretions. It has 
no appetite for our grief, pain, or terror, although it eagerly welcomes the behavioral surplus that leaches 
from our anguish. It is profoundly and infinitely indifferent to our meanings and motives. Trained on 
measurable action, it only cares that whatever we do is accessible to its ever-evolving operations of 
rendition, calculation, modification, monetization, and control. 

Although it is not murderous, instrumentarianism is as startling, incomprehensible, and new to the 
human story as totalitarianism was to its witnesses and victims. Our encounter with unprecedented power 
helps to explain why it has been difficult to name and know this new species of coercion, shaped in 
secret, camouflaged by technology and technical complexity, and obfuscated by endearing rhetoric. 
Totalitarianism was a political project that converged with economics to overwhelm society. 
Instrumentarianism is a market project that converges with the digital to achieve its own unique brand of 
social domination. 

It is not surprising, therefore, that instrumentarianism’s specific “viewpoint of observation” was 
forged in the controversial intellectual domain known as “radical behaviorism” and its antecedents in 
turn-of-the-century theoretical physics. In the remainder of this chapter, our examination of power in the 
time of surveillance capitalism pivots to this point of origin far from totalitarianism’s murder and 


mayhem. It takes us to laboratories and classrooms and the realms of thought spun by men who regarded 
freedom as a synonym for ignorance and human beings as distant organisms imprisoned in patterns of 
behavior beyond their own comprehension or control, such as ants, bees, or Stuart MacKay’s herds of elk. 


IV. The Other-One 


In a 1971 cover story, Time magazine described Burrhus Frederic “B. F.” Skinner as “an institution at 
Harvard... the most influential of living American psychologists, and the most controversial 
contemporary figure in the science of human behavior, adored as a messiah and abhorred as a menace.”=2 
Skinner spent most of his career in the Psychology Department at Harvard University, and some of my 
most vivid memories of graduate school are the times I spent with him in close debate. I admit that those 
conversations did little to alter our respective views, but they left me with an indelible sense of 
fascination with a way of construing human life that was—and is—fundamentally different from my own. 

As an academic psychologist, Skinner was famous for the ingenious tools and techniques he invented 
to study animal behavior—first in the ancient murky basement of Memorial Hall and later on the seventh 
floor of the newly constructed William James Hall—as well as the insights that he and his students 
developed into the shaping of that behavior: pigeons and levers, pigeons and pellets, pigeons on 
hotplates, rats in mazes. His early research broke new ground as he engineered variable “schedules of 
reinforcement” to produce detailed patterns of activity that were foreign to an animal’s original 
behavioral repertoire, work that he called “operant conditioning.” 

Inspired by his effort during World War II to transform a flock of pigeons into the conditioned 
intelligence for guided missiles (one that did not ultimately come to fruition), Skinner set out on a new 
path defined by the promise of behavioral engineering. In 1947 he told a symposium, “It is not a matter of 
bringing the world into the laboratory, but of extending the practices of an experimental science to the 
world at large. We can do this as soon as we wish to do it.”22 The missile project had cast the meaning of 
his experimental work “in a new light,” he reflected years later in his autobiography. “It was no longer 
merely an experimental analysis. It had given rise to a technology.” 

Skinner was eager to apply his laboratory insights to the world’s ills despite precious few grounds for 
his inferential leaps. As a public intellectual, he spent nearly seven decades trying to persuade the public 
that his radical behaviorism offered the principles of social organization necessary to defend civilization 
from cataclysm. He brashly extrapolated from the conduct of beleaguered animals to grand theories of 
social behavior and human evolution in books such as his 1948 “utopian” novel Walden Two and his 1971 
social philosophy Beyond Freedom & Dignity. In 1974 Skinner published About Behaviorism, another 
explanation of the radical behaviorist project, this time addressed to a general readership. It aimed to 
counter the opposition to his views that had grown even more virulent since the unusual—and, to many, 
repugnant—arguments advanced in Beyond Freedom & Dignity. He regarded such opposition as the 
result of an “extraordinary misunderstanding,” and he was tireless in his efforts to reverse public opinion. 
He believed that once people correctly understood his meaning, they would certainly agree with his 
message. 

In the very first pages of About Behaviorism, Skinner ignores the outrage generated by Beyond 
Freedom & Dignity, instead delving into behaviorism’s roots and its earliest theorists and practitioners. 
He places much of the blame for the antipathy toward behaviorism on the man who is widely regarded as 
its founder, John B. Watson.22 It was Watson who famously announced in 1913 the behaviorist point of 
view: “Psychology as the behaviorist views it is a purely objective experimental branch of natural 


science. Its theoretical goal is the prediction and control of behavior. Introspection forms no essential part 
of its methods.... The behaviorist... recognizes... no dividing line between man and brute.”2° But Watson 
turned out to be as much showman as scientist, and Skinner bitterly criticizes his extreme claims and 
“shortage of facts” that cast lasting doubt on radical behaviorism. 

Having identified Watson as the main author of the problem, Skinner then credits the work of Max 
Meyer, an obscure early-twentieth-century German-trained experimental psychologist who spent most of 
his career at the University of Missouri, as key to the solution. Meyer had pursued his doctoral studies at 
the University of Berlin, where his dissertation advisor, Max Planck, was destined to become one of the 
most celebrated physicists of all time. Planck insisted on the unity of the physical world and the 
discoverability of the natural laws that would disclose their secrets only through mathematical analysis, 
including the laws of human behavior.2’ “The outside world is something independent from man, 
something absolute,” Planck wrote. “The quest for the laws which apply to this absolute appeared... as 
the most sublime scientific pursuit in life.’22 Meyer carried Planck’s teaching into his quest for the 
principles that would finally elevate the study of human behavior to true scientific status. 

According to Skinner, Meyer succeeded in achieving the breakthrough that finally allowed psychology 
to assume its rightful place alongside the disciplines of physics, chemistry, and biology.22 Why did 
Skinner extol this work that had been largely ignored, even in its own time? Skinner singles out for 
special praise a 1921 textbook that bore the ominous-sounding title Psychology of the Other-One. It 
received scant attention when it was published—Meyer had written it primarily for his own students— 
and had since fallen into total obscurity.“2 Nonetheless, Skinner praised the book for establishing the 
epistemological and methodological foundations of modern behaviorism: “consider only those facts 
which can be objectively observed in the behavior of one person in its relation to his prior environmental 
history.” In Skinner’s formulation, Meyer’s book was a turning point, bravely combining psychology and 
physics in the quest for absolutes. It asserted the essence of the behaviorist’s point of view, where “the 
world within the skin of the Other-One loses its preferred status.”4 

The phrase that captured the new scientific perspective was the “Other-One.” Human behavior would 
yield to scientific research only if psychologists learned to view humans as others. This “viewpoint of 
observation” was an absolute requirement for an “objective science of human behavior” that ceased to 
confuse inner experience with external action.“? Central to this new viewpoint was his notion of the 
human being as organism. The human being is recast as an “it,” an “other,” a “they” of organisms: an 
“organism among organisms,” distinguishable from a lettuce, a moose, or an inchworm only in degree of 
complexity.“ A scientific psychology would restrict its interests to the social and therefore visible 
behaviors of this “organism as an organism.” It would be “the study of the life of the Other-One—but of 
his life in so far as it is of social significance rather than as it is of significance for himself.... We are 
studying the Other-One in preference to Our-Selves.”” 

The logical consequences of the new viewpoint necessitated a reinterpretation of the higher-order 
human experiences that we call “freedom” and “will.” Meyer echoed Planck in positing that “freedom of 
action in the animal world signifies the same that is meant by accidents in the world of physics.”“° Such 
accidents are simply phenomena for which there are insufficient information and understanding. And so it 
goes with freedom. The liberal idea of freedom persists in an inverse relationship to the growth of 
scientific knowledge, especially in the field of psychological science. Knowledge and freedom are 
necessarily adversaries. As Meyer wrote, “The Other-One’s conduct is free, uncaused, only in the same 
sense in which the issue of a disease, the outcome of a war, the weather, the crops are free and uncaused; 
that is, in the sense of general human ignorance of the particular causes of the particular outcome.” 

Decades later, this worldview would define the core of the controversial social philosophy espoused 


in Beyond Freedom & Dignity, in which Skinner argues that knowledge does not make us free but rather 
releases us from the illusion of freedom. In reality, he writes, freedom and ignorance are synonyms. The 
acquisition of knowledge is heroic in that it rescues us from ignorance, but it is also tragic because it 
necessarily reveals the impossibility of freedom. 

For Meyer and for Skinner, our attachments to notions such as freedom, will, autonomy, purpose, and 
agency are defense mechanisms that protect us from the uncomfortable facts of human ignorance. I think of 
Dickens’s Scrooge when he first encounters the doleful, chain-dragging ghost of his deceased partner 
Jacob Marley and denies the apparition, saying, “You may be an undigested bit of beef, a blot of mustard, 
a crumb of cheese, a fragment of an underdone potato.” So it is with freedom: an undigested bit of fear, a 
crumb of denial that, once metabolized, will dispel the apparition and deliver us to reality. The 
environment determines behavior, and our ignorance of precisely how it does so is the void that we fill 
with the fantasy of freedom. 

Meyer took great pains to insist that the significance of human inwardness—‘“soul,” “self,” “mind,” 
“consciousness”—is restricted to the subjective life of the individual. It can have no scientific value 
because it cannot be observed and measured: “We do not deny the soul, but we do not devote our time to 
it. We find enough, and more than enough, to do studying the body.” The soul is “your own business,” an 
intimate experience and irreducible mystery that are formally exempt from scientific inquiry: “Human 
societies can thus be understood as originating from natural laws, not in the sense of groups of souls tho, 
but in the sense of groups of organisms.” 

Meyer argued that the future of the social sciences and of civilization itself rested on this shift from 
soul to other, inside to outside, lived experience to observable behavior. The otherization of humanity 
was to be the road to a new kind of political liberation. History’s grim cavalcade of repression, torture, 
massacre, enslavement, and genocide had been prosecuted in the name of the domination of the human 
soul for the sake of religious or political power. From Meyer’s vantage point in the Missouri of 1921 
during those years after the Great War, his solution for an efficient and rational modernity must have felt 
like a matter of life and death: 


He whose interest is centered in souls thinks, when he has forced others to speak his prayer, pronounce his creed, kneel before his 
altar, that he has saved their souls, and fails to admit that he has merely forced their bodies.... Political terrorism, too, has its main and 
inexhaustible source in the human tendency to think of other beings, not as bodies open to scientific investigation, but as souls, as 
mysterious beings, to be governed either by magic, or, if magic fails as it naturally must, by torture and death.... Recall the horrors of 


the torture... of such courts as the Spanish inquisition or the witch-craft courts of the 17th century.... These atrocities were due to 


the fact that the judge was serving as a mind reader, and that the accused was regarded primarily as a soul.42 


In Meyer’s thinking, the shift in perspective from “the human being as a soul” to “the human being as 
an organism’ explained “why the trend of history is in the direction of democracy.” As science overtakes 
civilization, Meyer assumed a new global recognition of equality and democratic fellowship rooted in the 
basic fact of our overwhelming similarity as fellow organisms. The divisions that haunt society, politics, 
and economics based on class, wealth, liberality, race, and so on would become ridiculous: “In real 
world-wide human life, the differences among individuals are entirely swamped by their likenesses. To 
him who accepts the scientific view that human society is a group of organisms, it is an absurd 
proposition to divide... into... classes....”2 

Meyer believed that every social science that aspired to true scientific status would embrace the 
viewpoint of the Other-One—psychology, of course, but also sociology, economics, criminology, history, 
and the psychology of religion: “Christ going among his fellow men, an organism among organisms....”2! 


Otherizing would pave the way to a rational future, with its bleak satisfactions that resign humanity to the 
forfeit of freedom as the price of knowledge. 


V. Against Freedom 


Skinner’s commitment to the viewpoint of the Other-One was unshakable, and it is through his elaboration 
of this viewpoint that we can begin to grasp the essence of instrumentarian power. From the opening page 
of Skinner’s first book, The Behavior of Organisms, published in 1938, he sounds Meyer’s (and 
Planck’s) caution: freedom is merely ignorance waiting to be conquered. “Primitive systems of behavior” 
assign causality to “entities beyond man.” Equally inadequate are the “advanced systems of behavior” that 
ascribe control to vague fictions such as the “self” or “free will.” “The inner organism,” he wrote, “may 
in resignation be called free, as in the case of ‘free will,’ when no further investigation is held to be 
possible.”22 

Skinner called his work “radical behaviorism,” insisting that the only meaningful object of behavioral 
study was observation of action devoid of subjective attributions. That’s what made it radical. “Behavior 
is what an organism is doing—or more accurately what it is observed by another organism to be doing,” 
he declared in his first book’s opening pages. “Operant” behavior was his term for this active, observable 
“doing.” The vocabulary for rendering descriptions of operant behavior was to be purged of inwardness: 
an organism cannot be said to “see” but rather to “look toward.” Only such objective descriptions can 
render measurable behavioral facts that, in turn, lead to patterns and ultimately to the documentation of 
causal relationships between environment and behavior. 

Skinner published Science and Human Behavior in 1951, positing that all observation, even of one’s 
own behavior, must be enacted from the viewpoint of the Other-One. This discipline makes it possible to 
take almost anything as an object of behavioral analysis, including inferred behaviors such as “making 
choices” or “problem solving,” the very perspective that would later be amply exploited by the new 
discipline of behavioral economics: 


When a man controls himself, chooses a course of action, thinks out the solution to a problem, or strives toward an increase in self- 
knowledge, he is behaving. He controls himself precisely as he would control the behavior of anyone else—through the manipulation 


of variables of which behavior is a function. His behavior in so doing is a proper object of analysis, and eventually it must be 


accounted for with variables lying outside the individual himself.24 


In nearly every book and article, Skinner declared the truth that Planck taught Meyer and Meyer 
imparted to his students, the single truth that can be grasped only through the viewpoint of the Other-One: 
freedom is ignorance. The felt experience of free will is but a bit of undigested denial, produced by a 
lack of information about the actual determinants of behavior. Like Meyer and Planck before him, Skinner 
regarded freedom as an “accident,” arguing that the very notion of “accident” is illusory, a snapshot in 
time that reveals a lacuna waiting to be filled and eventually transformed by advancing knowledge into an 
expression of a lawful, predictable pattern. Under the behaviorist’s gaze, these lacunae of ignorance that 
we mistake for free will are queuing up for explanation, rather like someone who consigns his or her body 
to cryonic preservation in the hope of some future awakening and cure. 

In the most audacious of Skinner’s literary efforts, the extended philosophical essay published in 1971 
as Beyond Freedom & Dignity, he repeats: “There is no virtue in the accidental nature of an accident.” 
That entire work was trained on what Skinner continued to regard as the chief impediment to social 


progress: the conceptual confusion that cloaks our deepest ignorance in the sacred robes of freedom and 
dignity. Skinner argued that our allegiance to these lofty notions is simply the way we protect ourselves 
from the hard truths of “unsuspected controlling relations between behavior and environment.”2° They are 
a psychological “escape route” that slowly closes “as new evidences of the predictability of human 
behavior are discovered. Personal exemption from a complete determinism is revoked as a scientific 
analysis progresses... the achievements for which a person himself is to be given credit seem to approach 
zero... the behavior we admire is therefore the behavior we cannot yet explain.” 

Richard Herrnstein, one of Skinner’s most-accomplished students, later his colleague in Harvard’s 
Psychology Department, and a luminary of radical behaviorism, once explained to me that any action 
regarded as an expression of free will is simply one for which “the vortex of stimuli” that produced it 
cannot yet be adequately specified. We merely lack the means of observation and calculation. I was a 
twenty-three-year-old student, and the term was new and startling to me. I never forgot that conversation, 
perhaps because it came as close as any to summarizing the behaviorist’s conception of God. Indeed, 
there was a time when if you took the elevator to Skinner’s lab on the seventh floor of the Psychology 
Department, the first thing you would see was a sign reading “God is a VI,” a variable interval of 
behavioral reinforcement plucked from the vortex. 

In this view, “freedom” or “accidents” shrink as our developing capabilities in measurement and 
computation provide more information about the vortex of stimuli. Ignorance about human behavior is like 
a melting iceberg on a warming planet, destined to yield to the mounting heat as we invent the means and 
methods smart enough to first decipher and then control the vortex of stimuli that shapes each fact of 
human behavior. Skinner pointed to weather forecasting as the iconic example of this transformation from 
ignorance to law, just as Meyer had done decades earlier: 


The problems imposed by the complexity of a subject matter must be dealt with as they arise. Apparently hopeless cases often 
become manageable in time. It is only recently that any sort of lawful account of the weather has been possible.... Self-determination 


does not follow from complexity.... Difficulty in calculating the orbit of the fly does not prove capriciousness, though it may make it 


impossible to prove anything else.28 


VI. A Technology of Human Behavior 


Through six decades of academic and popular writing, Skinner would insist that “further investigation” is 
always possible. In the first pages of Beyond Freedom & Dignity he calls for a technological solution to 
ignorance: “We need to make vast changes in human behavior, and we cannot make them with the help of 
nothing more than physics or biology, no matter how hard we try... what we need is a technology of 
behavior... comparable in power and precision to physical and biological technology....”=2 

Skinner imagined technologies that would pervasively institutionalize the viewpoint of the Other-One 
as they observed, computed, analyzed, and automatically reinforced behavior to accomplish the “vast 
changes” that he believed were necessary. In this way the laws of human action would finally be 
illuminated so that behavior could be effectively predicted and shaped, just as other technologies had 
enabled physics and biology to change the world: “The difference is that the instruments and methods they 
use are of commensurate complexity. The fact that equally powerful instruments and methods are not 
available in the field of human behavior is not an explanation; it is only part of the puzzle.” 

Skinner concluded that the literature of freedom and dignity “stands in the way of further human 
achievement.”®! He argued that the missing puzzle piece holding back the urgent development of the 


“instruments and methods” essential for a technology of behavior was the stubborn allegiance to these 
antique notions among people determined to preserve “due credit” for their actions. The belief in 
“autonomous man” is a regressive source of resistance to a rational future, an “alternative explanation of 
behavior” that obstructs the advancement of society. 

The professor believed that humanity desperately needed a plan, and he imagined powerful new 
instruments that could engineer behavior in every domain. As early as 1953, he anticipated today’s 
digitally engineered casino environments, whose sophistication in the precise shaping of gamblers’ 
behavior has made them a testing ground for state security agencies and surveillance capitalists alike: 
“But with proper instrumentation it should be possible to improve upon established practices in all these 
fields. Thus gambling devices could be ‘improved’—/from the point of view of the proprietor—by 
introducing devices which would pay off on a variable-interval basis, but only when the rate of play is 
exceptionally high.” 

Technologies of behavioral engineering would not be restricted to “devices” but would also 
encompass organizational systems and procedures designed to shape behavior toward specific ends. In 
1953 Skinner anticipated innovations such as Michael Jensen’s incentive systems designed to maximize 
shareholder value and the “choice architectures” of behavioral economics designed to “nudge” behavior 
along a preferred path: “Schedules of pay in industry, salesmanship, and the professions, and the use of 
bonuses, incentive wages, and so on, could also be improved from the point of view of generating 
maximal productivity.”™ 

Skinner understood that the engineering of behavior risked violating individual sensibilities and social 
norms, especially concerns about privacy. In order to allay these anxieties he advised that observation 
must be unobtrusive, ideally remaining outside the awareness of the organism: “Behavior may also be 
observed with a minimum of interaction between subject and scientist, and this is the case with which one 
naturally tries to begin.”®2 But there would be challenges. New technologies of behavior would have to 
continually push the envelope of the public-private divide in order to access all the data relevant to 
behavioral prediction and control. In this he anticipated today’s rendition frontier as new detection 
systems plumb the depths of personalities and emotions: “But we are still faced with events which occur 
at the private level and which are important to the organism without instrumental amplification. How the 
organism reacts to these events will remain an important question, even though the events may some day 
be made accessible to everyone.”© 

Such conflicts would be resolved, Skinner reckoned, by the gradual retreat of privacy norms as they 
fall to the advance of knowledge: “The line between public and private is not fixed.” Like today’s 
surveillance capitalists, he was confident that the slow drip of technological invention would eventually 
push privacy to the margins of human experience, where it would join “freedom” and other troublesome 
illusions. All these would be replaced by the viewpoint of the Other-One embodied in new instruments 
and methods: “The boundary shifts with every discovery of a technique for making private events public. 
... The problem of privacy may, therefore, eventually be solved by technical advances.”®& 

Although privacy advocates and many other critics of surveillance capitalism are quick to appropriate 
Orwellian language in the search for meaning and metaphor that capture the sense of new menace, 
surveillance capital’s instrumentarian power is best understood as the precise antithesis of Orwell’s Big 
Brother. We now turn to this stark contrast, most vividly represented in the distinct conceptions of utopia 
that attach to each of these two species of power. 


VII. Two Utopias 


In the blood-soaked wake of World War II, both Skinner and the journalist and novelist George Orwell 
aimed curative, “utopian” novels at the mayhem of modernity’s diminished prospects produced by an 
incomprehensible scale of violence. Viewed from a distance, Skinner’s Walden Two, published in 1948, 
and Orwell’s 1984, released the following year, had much in common. Each elaborated a complete 
conception of a distinct logic of power, each imagined a society defined by the full flourishing of that 
power, and each was utopian from the point of view of the form of power that it described. However, 
their public reception could not have been more different: Walden Two was dismissed as a dystopian 
nightmare and ignored by the general public for more than a decade.®? Orwell’s 1984 was immediately 
canonized as a dystopian masterpiece and the distillation of the twentieth century’s worst nightmares. 

The two utopias have often been confused with each other in their content and aims: Time magazine’s 
1971 cover story on Skinner described Walden Two as raising “the specter of a 1984 Orwellian society 
that might really come to pass.” The great historian and literary critic Lewis Mumford once described 
Walden Two as a “totalitarian utopia” and a depiction of “hell,” but in fact these characterizations are a 
persistent and, as we shall see, dangerous confusion. Although both books have been described as 
depictions of totalitarianism, the forms of power that each describes are profoundly different. In most 
respects, they are precise opposites. 

Meyer’s prescription for modernity was founded on the scientific objectification of human experience 
and its reduction to observable measurable behavior. If we take that as our benchmark here, then Orwell’s 
utopia is the “before” case; it precedes Meyer as a nightmare of the prescientific compulsion to dominate 
the soul. Skinner’s utopia is “after” Meyer’s reimagining of modernity, as channeled from the great 
physicist, Planck. Walden Two is built on Meyer’s scientific viewpoint of observation, the Other-One, and 
represents the full flower of Meyer’s hope for a scientifically induced global harmony of organisms in 
which freedom is necessarily forfeit to knowledge. Orwell laid bare the disease, and Skinner asserted the 
antidote. 

The totalitarian power elaborated in 1984 is something wholly unprecedented, concocted in the 
twentieth century from the collision of industrialism and despair, a form utterly new to the human story. 
Orwell did more than simply fictionalize and extrapolate the totalitarian project. He sounded an alert by 
drawing a terrifying line of consequence from the recent German past and persistent Soviet present to an 
imagined but all-too-possible future. His genius was to craft a story that embodied totalitarianism’s 
essence: the ruthless insistence upon the absolute possession of each individual human being, not as a 
distant other known only by its behavior, but rather from the inside out. 

Big Brother’s vigilance is not restricted to the grand continents of armies and statecraft or the 
observable flows of bodies and crowds. Big Brother is a panvasive consciousness that infects and 
possesses each individual soul, displacing all attachments once formed in romantic love and good 
fellowship. The essence of its operation is not simply that it knows every thought and feeling but rather 
the ruthless tenacity with which it aims to annihilate and replace unacceptable inward experience. “We 
are not content with negative obedience, nor even with the most abject submission,” the cunning henchman 
O’Brien tells the rebellious Winston: 


We shall crush you down to the point from which there is no coming back.... When finally you surrender to us, it must be of your 
own free will. We do not destroy the heretic because he resists us: so long as he resists us we never destroy him. We convert him, 


we capture his inner mind, we reshape him.... We bring him over to our side, not in appearance, but genuinely, heart and soul. We 


make him one of ourselves before we kill him, 22 


Ultimately, as every reader knows, Winston’s stubborn soul is successfully “engineered.” Orwell’s 


chilling final passages fulfill the life of that dry seed planted at the turn of the century in Italy’s 
impoverished soil and nourished by war, deprivation, and humiliation to flower in the nightmare of Nazi 
Germany and the apocalypse of Stalin’s Russia, finally to bear fruit in Orwell’s imagination: a testament 
for all time to what Mussolini had called the “ferocious totalitarian will” and the souls on which it feeds. 
Winston basks in serene elation, “his soul white as snow.... He had won the victory over himself. He 
loved Big Brother.” 

In contrast, Walden Two was not intended as a warning but rather as the antidote to totalitarianism and, 
more broadly, a practical recipe for the challenges of rebuilding Western societies after the war. Skinner 
understood his utopia as a methodological cure for the nightmare of crushed souls, a cure that, he insisted, 
was superior to any of the conventional political, economic, or spiritual remedies on offer. He scoffed at 
the notion that “democracy” held the solution because it is a political system that merely perpetrates the 
illusion of freedom while impeding the dominion of science. The promise of the “free market” as the 
curative for postwar society was an equally empty dream, he believed, because it rewards destructive 
competitiveness between people and classes. Skinner also rejected existentialism’s new humanism, 
seeing it as a breeding ground for passivity, and he regarded religion as the worst cure of all, enshrining 
ignorance and crippling the advance of science. 

Skinner’s cure was different and unique: a utopia of technique that promised a future of social equality 
and dispassionate harmony founded on the viewpoint of the Other-One, the “organism among organisms,” 
as the object of “behavioral engineering.” It is the future of Meyer’s dreams, in which Frazier, the 
founder-leader of the fictional Walden Two and Skinner’s outspoken avatar, describes that ideal 
community as a “superorganism” that can be shaped and controlled “as smoothly and efficiently as 
champion football teams.” 

Skinner’s utopia was meant to illustrate the possibility of a successful social order that transcends the 
use of force and rejects the need to dominate human souls. The Walden Two community equally disdains 
the practices of democratic politics and representative government. Its laws are derived from a science of 
human behavior, specifically Skinner’s own radical behaviorism, founded on the physicist’s ideal of the 
Other-One. His utopia was a vehicle for other ambitions as well, intended to illustrate the behavioral 
solutions that are essential for improvement in every domain of modern life: the nuclear threat, pollution, 
population control, urban growth, economic equality, crime, education, health care, the development of 
the individual, effective leisure. It aimed to cultivate “the good life,” for which all the ideals of a liberal 
society—freedom, autonomy, privacy, a people’s right to self-rule—must be forfeit. 

Walden Two’s fictional format provided the cover that Skinner needed to extrapolate from Meyer’s 
methodological principles of otherness and his own research on animal behavior to a utopian community 
in which behavior has replaced the human spirit as the locus of control. Frazier laments that people “have 
been kept in their places,” not just by external forces, “but much more subtly by a system of beliefs 
implanted within their skins. It’s sometimes an almost hopeless task to take the shackles off their souls, 
but it can be done.... You can’t in the long run enforce anything. We don’t use force! All we need is 
adequate behavioral engineering.” 

These two utopias reflect two distinct species of power, and each novel was intent on rescuing the 
future from the twentieth-century nightmare of the soul. Orwell was able to draw on the recent past, but 
Skinner imagined a future that he would not live long enough to enjoy. If surveillance capitalism and its 
instrumentarian power continue to flourish, then it is we who may see the Walden Two vision realized, as 
freedom falls to others’ knowledge—but now in the service of others’ wealth and power. 

Skinner’s vision is brought to life in the relentless pursuit of surveillance capitalism’s economic 
imperatives and the ubiquitous digital apparatus that surveillance capitalism creates and harnesses to its 


novel aims. Instrumentarian power bends the new digital apparatus—continuous, autonomous, 
omnipresent, sensate, computational, actuating, networked, internet-enabled—to the interests of the 
surveillance capitalist project, finally fulfilling Skinner’s call for the “instruments and methods” of “a 
behavioral technology comparable in power and precision to physical and biological technology.” The 
result is a panvasive means of behavioral modification whose economies of action are designed to 
maximize surveillance revenues. 

Until the rise of surveillance capitalism, the prospect of instrumentarian power was relegated to a 
gauzy world of dream and delusion. This new species of power follows the logic of Planck, Meyer, and 
Skinner in the forfeit of freedom for knowledge, but those scientists each failed to anticipate the actual 
terms of this surrender. The knowledge that now displaces our freedom is proprietary. The knowledge is 
theirs, but the lost freedom belongs solely to us. 

With this origin story in hand, Chapter 13 turns to a close examination of instrumentarian power as it 
inscribes a sharp new asymmetry on the human community: the knowledge to which we sacrifice our 
freedom is constructed to advance surveillance capitalists’ commercial interests, not our own. This is a 
stark departure from the technical origins of the apparatus in MacKay’s principles of telemetry, which 
traded animals’ freedom for the sake of scientific knowledge intended to benefit the animals themselves. 
Instead, surveillance capitalism’s behavioral market regime finally has at its disposal the instruments and 
methods that can impose Skinner’s technology of behavior across the varied domains of everyday life 
right down to our depths, now conceived as capital’s global laboratory. 


CHAPTER THIRTEEN 


BIG OTHER AND THE RISE OF 
INSTRUMENTARIAN POWER 


He was their servant (some say he was blind), 
Who moved among their faces and their things: 
Their feeling gathered in him like a wind 
And sang. They cried “It is a God that sings.” 


—W. H. AUDEN 
SONNETS FROM CHINA, VII 


I. Instrumentarianism as a New Species of Power 


Surveillance capitalism is the puppet master that imposes its will through the medium of the ubiquitous 
digital apparatus. I now name the apparatus Big Other: it is the sensate, computational, connected puppet 
that renders, monitors, computes, and modifies human behavior. Big Other combines these functions of 
knowing and doing to achieve a pervasive and unprecedented means of behavioral modification. 
Surveillance capitalism’s economic logic is directed through Big Other’s vast capabilities to produce 
instrumentarian power, replacing the engineering of souls with the engineering of behavior. 

Instrumentarian power cultivates an unusual “way of knowing” that combines the “formal 
indifference” of the neoliberal worldview with the observational perspective of radical behaviorism (see 
Figure 4 here). Thanks to Big Other’s capabilities, instrumentarian power reduces human experience to 
measurable observable behavior while remaining steadfastly indifferent to the meaning of that experience. 
I call this new way of knowing radical indifference. It is a form of observation without witness that 
yields the obverse of an intimate violent political religion and bears an utterly different signature of 
havoc: the remote and abstracted contempt of impenetrably complex systems and the interests that author 
them, carrying individuals on a fast-moving current to the fulfillment of others’ ends. What passes for 
social relations and economic exchange now occurs across the medium of this robotized veil of 
abstraction. 

Instrumentarianism’s radical indifference is operationalized in Big Other’s dehumanized methods of 
evaluation that produce equivalence without equality. These methods reduce individuals to the lowest 
common denominator of sameness—an organism among organisms—despite all the vital ways in which 
we are not the same. From Big Other’s point of view we are strictly Other-Ones: organisms that behave. 


Big Other encodes the viewpoint of the Other-One as a global presence. There is no brother here of any 
kind, big or little, evil or good; there are no family ties, however grim. There is no relationship between 
Big Other and its otherized objects, just as there was no relationship between B. F. Skinner’s “scientists 
and subjects.” There is no domination of the soul that displaces all intimacy and attachment with terror— 
far better to let a multitude of relationships bloom. Big Other does not care what we think, feel, or do as 
long as its millions, billions, and trillions of sensate, actuating, computational eyes and ears can observe, 
render, datafy, and instrumentalize the vast reservoirs of behavioral surplus that are generated in the 
galactic uproar of connection and communication. 

In this new regime, objectification is the moral milieu in which our lives unfold. Although Big Other 
can mimic intimacy through the tireless devotion of the One Voice—Amazon-Alexa’s chirpy service, 
Google Assistant’s reminders and endless information—do not mistake these soothing sounds for anything 
other than the exploitation of your needs. I think of elephants, that most majestic of all mammals: Big 
Other poaches our behavior for surplus and leaves behind all the meaning lodged in our bodies, our 
brains, and our beating hearts, not unlike the monstrous slaughter of elephants for ivory. Forget the cliché 
that if it’s free, “You are the product.” You are not the product; you are the abandoned carcass. The 
“product” derives from the surplus that is ripped from your life. 

Big Other finally enables the universal technology of behavior that, as Skinner, Stuart MacKay, Mark 
Weiser, and Joe Paradiso each insisted, accomplishes its aims quietly and persistently, using methods that 
intentionally bypass our awareness, disappearing into the background of all things. Recall that 
Alphabet/Google’s Eric Schmidt provoked uproar in 2015 when in response to a question on the future of 
the web, he said, “The internet will disappear.” What he really meant was that “The internet will 
disappear into Big Other.” 

Thanks to Big Other’s capabilities, instrumentarian power aims for a condition of certainty without 
terror in the form of “guaranteed outcomes.” Because it does not claim our bodies for some grotesque 
regime of pain and murder, we are prone to undervalue its effects and lower our guard. Instead of death, 
torture, reeducation, or conversion, instrumentarianism effectively exiles us from our own behavior. It 
severs our insides from our outsides, our subjectivity and interiority from our observable actions. It lends 
credibility to the behavioral economists’ hypothesis of the frailty of human reason by making it so, as 
otherized behavior takes on a life of its own that delivers our futures to surveillance capitalism’s aims 
and interests. 

In an instrumentarian utopia, Big Other simulates the vortex of stimuli, transforming “natural selection” 
into the “unnatural selection” of variation and reinforcement authored by market players and the 
competition for surveillance revenues. We may confuse Big Other with the behaviorist god of the vortex, 
but only because it effectively conceals the machinations of surveillance capital that are the wizard 
behind the digital curtain. The seductive voice crafted on the yonder side of this veil—Google, is that 
you?—gently nudges us along the path that coughs up the maximum of behavioral surplus and the closest 
approximation to certainty. Do not slumber in this opiated fog at the network’s edge. That knowing voice 
is underwritten by the aims and rules of the very place we once hoped to flee, with its commercialized 
rituals of competition, contempt, and humiliation. Take one wrong step, one deviation from the path of 
seamless frictionless predictability, and that same voice turns acid in an instant as it instructs “the 
vehicular monitoring system not to allow the car to be started.” 

Under the regime of instrumentarian power, the mental agency and self-possession of the right to the 
future tense are gradually submerged beneath a new kind of automaticity: a lived experience of stimulus- 
response-reinforcement aggregated as the comings and goings of mere organisms. Our conformity is 
irrelevant to instrumentarianism’s success. There is no need for mass submission to social norms, no loss 


of self to the collective induced by terror and compulsion, no offers of acceptance and belonging as a 
reward for bending to the group. All of that is superseded by a digital order that thrives within things and 
bodies, transforming volition into reinforcement and action into conditioned response. 

In this way instrumentarian power produces endlessly accruing knowledge for surveillance capitalists 
and endlessly diminishing freedom for us as it continuously renews surveillance capitalism’s domination 
of the division of learning in society. False consciousness is no longer produced by the hidden facts of 
class and their relation to production but rather by the hidden facts of instrumentarian power’s command 
over the division of learning in society as it usurps the rights to answer the essential questions: Who 
knows? Who decides? Who decides who decides? Power was once identified with the ownership of the 
means of production, but it is now identified with ownership of the means of behavioral modification that 
is Big Other. 


IT. A Market Project of Total Certainty 


Big Other and the instrumentarian power that it enables fulfill Skinner’s vision for humankind. As early as 
1948 in Walden Two, Skinner had pined for the new measurement and computational capabilities that 
would unlock the mysteries of the vortex of stimuli and illuminate those acts of ignorance that we 
foolishly value as free will. “I didn’t say that behavior is always predictable, any more than the weather 
is always predictable,” says Frazier, Walden Two’s protagonist. “There are often too many factors to be 
taken into account. We can’t measure them all accurately, and we couldn’t perform the mathematical 
operations needed to make a prediction if we had the measurements.”+ 

It was Skinner’s hard destiny to foresee the possibility of instrumentarian power and _ its 
operationalization in Big Other before the instruments existed to fulfill that vision. His lab had always 
been a fantasy world of engineering innovations for his behavioral experiments: mazes and boxes for 
animal conditioning, measurement tools, and recording devices. A fully operational technology of 
behavior was the prize that would elude Skinner in his lifetime, a source of frustration that was palpable 
in every article and lecture right to the end. 

Though confident that science would eventually overcome the practical challenges of a behavioral 
technology, Skinner was more troubled by the cultural impediments to a human science of behavioral 
prediction and control founded on the viewpoint of the Other-One. He resented the friction produced by 
human beings in their stubborn attachment to the values and ideals of freedom inherited from the 
eighteenth-century philosophers, and he equally despised the existential project of the postwar 
philosophies that planted authenticity, free will, and autonomous action at the heart of second-modernity 
yearning. 

In his penultimate essay, written in 1990, barely three years before his death, Skinner mourned the 
prospects of behavioral prediction as the foundation of a new society built on scientific certainty: “To say 
that a person is simply a place in which something happens seems even more threatening when it raises 
questions about what we are likely to do rather than what we have done.”? In those last years he seemed 
more resigned to the tenacity of human friction and its stubborn allegiance to something like free will, his 
voice less outrageous and aggressive than the author of Beyond Freedom & Dignity, two decades earlier. 
Anger and contempt had shaded into weariness and despair with his final reflections: 


It may be said that this is a discouraging view of human behavior and that we shall be more likely to do something about the future if 


we continue to believe that our destiny is in our hands. That belief has survived for many centuries and has led to remarkable 


achievements, but the achievements were only the immediate consequences of what was done. We now know that other 
consequences have followed and that they are threatening our future. What we have done with our destiny may not be a testament 


we wish to leave to the world.3 


In our time of surveillance exceptionalism, as astonishment succumbs to helplessness and resignation, 
the resistance that Skinner lamented appears to be waning. The belief that we can choose our destiny is 
under siege, and, in a dramatic reversal, the dream of a technology of behavioral prediction and control— 
for which Skinner had endured such public scorn—is now a flourishing fact. This prize now attracts 
immense capital, human genius, scientific elaboration, governmental protection, ecosystems of 
institutionalization, and the glamour that always has and always will attach to power. 

The rise of instrumentarianism follows the path of “illuminating events” that, as Hannah Arendt writes, 
reveal “a beginning in the past which had hitherto been hidden.” It is in the nature of instrumentarian 
power to operate remotely and move in stealth. It does not grow through terror, murder, the suspension of 
democratic institutions, massacre, or expulsion. Instead, it grows through declaration, self-authorization, 
rhetorical misdirection, euphemism, and the quiet, audacious backstage moves specifically crafted to 
elude awareness as it replaces individual freedom with others’ knowledge and replaces society with 
certainty. It does not confront democracy but rather erodes it from within, eating away at the human 
capabilities and self-understanding required to sustain a democratic life. 

The narratives of Facebook’s experimental maneuvers, Pokémon Go’s prototype of a behavioral- 
futures-market—dominated society, and the endless examples of digital innovation crushed under the heel 
of the surveillance creed may be as close as we get to a public announcement of history-illuminating 
change that moves through us and among us, irreversibly altering life as we have known it. There is no 
violence here, only the steady displacement of the will to will that has been embodied in self- 
determination, expressed in the first-person voice, and nourished in the kind of sanctuary that depends 
upon the possibility of private life and the promise of public freedom. 

Instrumentarian power, like Goethe’s Faust, is morally agnostic. The only moral imperative here is 
distilled from the point of view of a thin utopian gruel. If there is sin, it is the sin of autonomy: the 
audacity to reject the flows that herd us all toward predictability. Friction is the only evil. Obstruction in 
law, action, or rhetoric is simply reactionary. The norm is submission to the supposed iron laws of 
technological inevitability that brook no impediment. It is deemed only rational to surrender and rejoice 
in new conveniences and harmonies, to wrap ourselves in the first text and embrace a violent ignorance of 
its shadow. 

Totalitarianism was a transformation of the state into a project of total possession. Instrumentarianism 
and Big Other signal the transformation of the market into a project of total certainty, an undertaking that is 
unimaginable outside the digital milieu, but also unimaginable outside the logic of accumulation that is 
surveillance capitalism. This new power is the spawn of an unprecedented convergence: the surveillance 
and actuation capabilities of Big Other in combination with the discovery and monetization of behavioral 
surplus. It is only in the context of this convergence that we can imagine economic principles that 
instrumentalize and control human experience to systematically and predictably shape behavior toward 
others’ profitable ends. 

Instrumentarian power operates from the vantage point of the Other-One to reduce human persons to 
the mere animal condition of behavior shorn of reflective meaning. It sees only organisms bent to serve 
the new laws of capital now imposed on all behavior. Arendt anticipated the destructive potential of 
behaviorism decades ago when she lamented the devolution of our conception of “thought” to something 
that is accomplished by a “brain” and is therefore transferable to “electronic instruments”: 


The last stage of the laboring society, the society of jobholders, demands of its members a sheer automatic functioning, as though 
individual life had actually been submerged in the over-all life process of the species and the only active decision still required of the 
individual were to let go, so to speak, to abandon his individuality, the still individually sensed pain and trouble of living, and acquiesce 
in a dazed, “tranquilized,” functional type of behavior. 

The trouble with modern theories of behaviorism is not that they are wrong but that they could become true, that they actually are 
the best possible conceptualization of certain obvious trends in modern society. It is quite conceivable that the modern age—which 


began with such an unprecedented and promising outburst of human activity—may end in the deadliest, most sterile passivity history 


has ever known.2 


Is this to be our home: the automation of the self as the necessary condition of the automation of 
society, and all for the sake of others’ guaranteed outcomes? 


III. This Century’s Curse 


One place to begin our reckoning with this question is in Arendt’s “Concluding Remarks” in the first 
edition of The Origins of Totalitarianism, words that have haunted me since I first read them many years 
ago. They were written at a time when totalitarianism had been defeated in Europe but remained 
unchallenged in Stalin’s USSR. It was a time when much of the world was united in the urgency of 
understanding and remembering, not only as testament but also as a vaccine against future terror. 

Arendt’s reflections summarize not only totalitarianism’s “futility” and “ludicrousness” but also her 
sense of the “disturbing relevance of totalitarian regimes.” She warned that totalitarianism could not be 
written off as an accidental turn toward tragedy but rather that it must be seen as “deeply connected with 
the crisis of this century.” She concluded, “The fact is that the true problems of our time cannot be 
understood, let alone solved, without acknowledgement that totalitarianism became this century’s curse 
only because it so terrifyingly took care of its problems.”° 

Now imagine, decades hence, another thinker meditating on the “disturbing relevance” of 
instrumentarian power, observing that “the true problems of our time cannot be understood, let alone 
solved, without acknowledgement that instrumentarianism became this century’s curse only because it so 
terrifyingly took care of its problems.” 

What problems? I have argued that surveillance capitalism and its instrumentarian power feed on the 
volatile “conditions of existence” that I summarize as the “collision.” Surveillance capitalism offers 
solutions to individuals in the form of social connection, access to information, time-saving convenience, 
and, too often, the illusion of support. These are the resources of the first text. More significantly, it offers 
solutions to institutions in the form of omniscience, control, and certainty. The idea here is not to heal 
instability—the corrosion of social trust and its broken bonds of reciprocity, dangerous extremes of 
inequality, regimes of exclusion—but rather to exploit the vulnerabilities produced by these conditions. 

It is useful to note that despite the much-touted social advantages of always-on connection, social trust 
in the US declined precipitously during the same period that surveillance capitalism flourished. 
According to the US General Social Survey’s continuous measurement of “interpersonal trust attitudes,” 
the percentage of Americans who “think that most people can be trusted” remained relatively steady 
between 1972 and 1985. Despite some fluctuations, 46 percent of Americans registered high levels of 
interpersonal trust in 1972 and nearly 50 percent in 1985. As the neoliberal disciplines began to bite, that 
percentage steadily declined to 34 percent in 1995, just as the public internet went live. The late 1990s 
through 2014 saw another period of steady and decisive decline to only 30 percent. 


Societies that display low levels of interpersonal trust also tend to display low levels of trust toward 
legitimate authority; indeed, levels of trust toward the government have also declined substantially in the 
US, especially during the decade and a half of growing connectivity and the spread of surveillance 
capitalism. More than 75 percent of Americans said that they trusted the government most or all of the 
time in 1958, about 45 percent in 1985, close to 20 percent in 2015, and down to 18 percent in 2017.8 
Social trust is highly correlated with peaceful collective decision making and civic engagement. In its 
absence, the authority of shared values and mutual obligations slips away. The void that remains is a loud 
signal of societal vulnerability. Confusion, uncertainty, and distrust enable power to fill the social void. 
Indeed, they welcome it. 

In the age of surveillance capitalism it is instrumentarian power that fills the void, substituting 
machines for social relations, which amounts to the substitution of certainty for society. In this imagined 
collective life, freedom is forfeit to others’ knowledge, an achievement that is only possible with the 
resources of the shadow text. 

The private institutions of capital led the way in this ambitious reformation of collective life and 
individual experience, but they found necessary support from public institutions, especially as the 
declaration of a “war on terror” legitimated every inclination to enshrine machine-produced certainty as 
the ultimate solution to societal uncertainty. These mutual affinities assured that instrumentarian power 
would not be a stepchild but rather an equal partner or even, with increasing regularity, the lord and 
master upon whom the state depends in its quest for “total awareness.” 

That instrumentarian power is regarded as the certain solution to uncertain societal conditions is 
evident in the ways in which it is called into action by the state. The varied and complex institutional 
patterns produced by these interactions are a crucial frontier for study and democratic debate. My aim 
right now is simply to point out a few examples that illustrate the state’s continuous demands for the 
intensification of surveillance capitalism’s production of instrumentarian power—expressed in the growth 
and elaboration of Big Other—as the preferred solution to social breakdown, mistrust, and uncertainty. 
Although we have become desensitized to a seemingly endless train of such examples, it is important to 
recognize that in these entanglements, state and market institutions demonstrate a shared commitment to a 
relentless drive toward guaranteed outcomes. Their mutual affinities can help us define the problem that 
threatens to make instrumentarian power our century’s curse. 

Unsurprisingly, instrumentarian power is consistently called into action as a solution, if not the 
solution, to the threat of terrorism. Acts of terror reject the authority of civilizational norms and reveal the 
impossibility of society without mutual trust. Governments now turn to instrumentarian power as the 
solution to this new source of societal uncertainty, demanding the certainty machines that promise direct, 
reliable means of detection, prediction, and even the automatic actuation of countermeasures. 

During the sixteen years of the Bush and Obama administrations, “progress in information technology” 
was understood as the “most effective response” to threat. Peter Swire observes that public officials 
“know that the private sector is developing many new techniques for collecting and processing data and 
making decisions based on that data.” The consequence is “a large and ongoing shift toward information- 
intensive strategies” that avail themselves of these market capabilities.2 

This status quo was interrupted in 2013, when Edward Snowden revealed the hidden complicity 
between state security agencies and the tech companies. In the backlash that followed, surveillance 
capitalists confronted new public relations challenges in what they portrayed as an uneasy or even 
unwilling alliance between private power and state security needs. Nevertheless, new terrorist threats 
invariably orient public officials toward the intensification and deployment of Big Other and the 
instrumentarian power it signifies. However, their ability to access this immense power is fraught with 


tension. It is not simply theirs to command. They must work, at least in part, through the surveillance 
capitalists. 

For example, in the aftermath of the December 2015 terror attacks in Paris, President Obama, US 
legislators, and public officials around the world exhorted the tech companies, especially Google, 
Facebook, and Twitter, to identify and remove terrorist content. The companies were reportedly reluctant 
to be, or at least to be perceived as, “tools of government.”!2 Journalists noted that public officials 
developed “workarounds” aimed at achieving access to instrumentarian power without imposing new 
burdens on the companies’ public standing. For example, a government agency could assert that offending 
online content violates the internet company’s terms of service, thus initiating the quick removal of 
offending material “without the paper trail that would go with a court order.” Similarly, Google expanded 
its “trusted flagger” program, through which officials and others could identify problematic content for 
immediate action. 

The companies responded with their own initiatives. Eric Schmidt suggested new instruments, 
including a “spell check for hate,” to target and eliminate terrorist accounts, remove content before it 
spreads, and accelerate the dissemination of counter-messages.* Top Obama administration officials 
endorsed that prospect on a pilgrimage to Silicon Valley in January 2016 for a “terror summit” with tech 
leaders. The agenda included discussions on how to disrupt terror activities on the internet, amplify 
alternative content, disrupt paths to radicalization, and enable security agencies to prevent attacks.° A 
White House briefing memo encouraged the companies to develop a “radicalism algorithm” that would 
digest social media and other sources of surplus to produce something comparable to a credit score, but 
aimed at evaluating the “radicalness” of online content./4 

The turn to instrumentarian power as the solution to uncertainty is not restricted to the US government. 
Terrorism triggers similar responses in Germany, France, the UK, and around the world. After the 2016 
attack on a Berlin Christmas market, German officials announced plans to require suspected extremists to 
wear electronic tags for perpetual tracking® In 2017 surveillance capitalists, including Facebook, 
Microsoft, Google, and Twitter, established the Global Internet Forum to Counter Terrorism. The 
objective was to tighten the net of instrumentarian power through “collaboration on engineering solutions 
to the problem of sharing content classification techniques,” “counterspeech initiatives,” and a shared 
database of “unique digital fingerprints” for violent terrorist imagery to accelerate the identification of 
terrorist accounts.1® The 2017 joint report of five countries—Australia, Canada, New Zealand, the United 
Kingdom, and the United States—included four key commitments, the very first of which was engagement 
with the internet companies to address online terrorism activities and to support the industry forum led by 
Google and Facebook.” That year, the European Council announced its expectation that “industry” would 
live up to its responsibility “to develop new technology and tools to improve the automatic detection and 
removal of content that incites to terrorist acts.”!8 Meeting in Hamburg in 2017, the G20 countries vowed 
to work with the internet companies, insisting on the need for better instruments to filter, detect, and 
remove content, and “encouraging” the industry to invest in the technology and human capital able to 
detect and eliminate terrorist activity online./® 

There are other emerging configurations of instrumentarian and state power. For example, former US 
Director of National Intelligence James Clapper told Congress in 2016 that the intelligence services might 
use the “internet of things” for “identification, surveillance, monitoring, location tracking, and targeting 
for recruitment, or to gain access to networks or user credentials.”22 Indeed, a research report from 
Harvard’s Berkman Klein Center for Internet & Society concluded that surveillance capitalism’s wave of 
“smart” appliances and products, networked sensors, and the “internet of things” would open up 
“numerous avenues for government actors to demand access to real-time and _ recorded 


communications.”2! 


That “smart” and “connected” signal new channels for commercial and government surveillance is 
neither conjecture nor limited to federal intelligence agencies. In a 2015 murder case, police used data 
from a “smart” utility meter, an iPhone 6s Plus, and audio files captured by an Amazon Echo device to 
identify a suspect.** In 2014 data from a Fitbit wristband were used ina personal injury case, and in 2017 
police used data froma pacemaker to charge a man with arson and insurance fraud.22 

In the US, local law enforcement has joined the queue of institutions seeking access to instrumentarian 
power. Surveillance-as-a-service companies eagerly sell their wares to local police departments also 
determined to find a shortcut to certainty. One startup, Geofeedia, specializes in detailed location tracking 
of activists and protesters, such as Greenpeace members or union organizers, and the computation of 
individualized “threat scores” using data drawn from social media. Law-enforcement agencies have been 
among Geofeedia’s most prominent clients.24 When the Boston Police Department announced its interest 
in joining this roster in 2016, the city’s police commissioner described to the Boston Globe his belief in 
machine certainty as the antidote to social breakdown: “The attack... on the Ohio State University campus 
is just the latest illustration of why local law-enforcement authorities need every tool they can muster to 
stop terrorism and other violence before it starts.”22 An ACLU attorney countered that the government is 
using tech companies “to build massive dossiers on people” based on nothing more than their 
constitutionally protected speech.2° Another, more prominent surveillance-as-a-service company, Palantir, 
once touted by Bloomberg Businessweek as “the war on terror’s secret weapon,” was found to be ina 
secret collaboration with the New Orleans Police Department to test its “predictive policing” technology. 
Palantir’s software not only identified gang members but also “traced people’s ties to other gang 
members, outlined criminal histories, analyzed social media, and predicted the likelihood that individuals 
would commit violence or become a victim.” ~ 


IV. The China Syndrome 


It is now possible to imagine one logical conclusion of this trend toward the substitution of certainty for 
society as the Chinese government develops a comprehensive “social credit” system described by one 
China scholar as the “core” of China’s internet agenda. The aim is “to leverage the explosion of personal 
data... in order to improve citizens’ behavior.... Individuals and enterprises are to be scored on various 
aspects of their conduct—where you go, what you buy and who you know—and these scores will be 
integrated within a comprehensive database that not only links into government information, but also to 
data collected by private businesses.”22 

The system tracks “good” and “bad” behavior across a variety of financial and social activities, 
automatically assigning punishments and rewards to decisively shape behavior toward “building 
sincerity” in economic, social, and political life: “The aim is for every Chinese citizen to be trailed by a 
file compiling data from public and private sources... searchable by fingerprints and other biometric 
characteristics.” 

Although China’s social credit vision is invariably described as “digital totalitarianism’ and is often 
compared to the world of Orwell’s 1984, it is better understood as the apotheosis of instrumentarian 
power fed by public and private data sources and controlled by an authoritarian state. The accounts of its 
pilot programs describe powerful examples of surveillance capitalism’s economies of action and the 
intricate construction of superscale means of behavior modification. The aim is the automation of society 
through tuning, herding, and conditioning people to produce preselected behaviors judged as desirable by 


the state and thus able to “preempt instability,” as one strategic studies expert put it.22 In other words, the 
aim is to achieve guaranteed social rather than market outcomes using instrumentarian means of 
behavioral modification. The result is an emergent system that allows us to peer into one version of a 
future defined by a comprehensive fusion of instrumentarian and state power. 

China’s vision is intended as the solution to its own unique version of the curse of social dissolution. 
Writing in Foreign Policy, journalist Amy Hawkins explains that China’s pandemic of social distrust is 
the problem to which the social credit system is addressed as the cure: “To be Chinese today is to live in 
a society of distrust, where every opportunity is a potential con and every act of generosity a risk of 
exploitation.”2! A fascinating empirical study of social trust in contemporary China actually finds high 
levels of “in-group” trust but discovers that these are correlated with negative health outcomes. The 
conclusion is that many Chinese trust only the people who are well-known to them. All other relationships 
are regarded with suspicion and anxiety, with obvious consequences for social trust as well as well- 
being.“ This rampant distrust, typically assigned to the traumas of rapid modernization and the shift to a 
quasi-capitalist economy, is also the legacy of Chinese totalitarianism. The Chinese Communist Party 
dismantled traditional domains of affiliation, identity, and social meaning—family, religion, civil society, 
intellectual discourse, political freedom—recalling Arendt’s description of the “atomization” that 
destroys bonds of trust.22 As Hawkins writes, “But rather than promoting the organic return of traditional 
morality to reduce the gulf of distrust, the Chinese government has preferred to invest its energy in 
technological fixes... and it’s being welcomed by a public fed up of not knowing who to trust... in part 
because there’s no alternative.”24 The Chinese government intends to commandeer instrumentarian power 
to replace a broken society with certain outcomes. 

In 2015 the Chinese central bank announced a pilot project in which the top e-commerce companies 
would pioneer the data integration and software development for personal credit scoring. Among the 
largest of the pilots was Alibaba’s Ant Financial and its “personal credit scoring” operation, “Sesame 
Credit.” The Sesame Credit system produces a “holistic” rating of “character” with algorithmic learning 
that goes far beyond the timely payment of bills and loans. Algorithms evaluate and rank purchases (video 
games versus children’s books), education degrees, and the quantity and “quality” of friends. One 
reporter’s account of her Sesame Credit experience warns that the algorithm veers into “voodoo,” 
considering the credit scores of her social contacts, the car she drives, her job, school, and a host of 
unspecified behavioral variables that supposedly “correlate with good credit.” The shadow text remains 
out of reach, and users are left to guess how to improve their scores, including shedding friends with low 
scores and bulking up on high-scoring individuals who, some believe, can boost one’s own rank. 

The company’s CEO boasts that the scoring system “will ensure that the bad people in society don’t 
have a place to go, while good people can move freely and without obstruction.” Those with high scores 
receive honors and rewards from Sesame Credit’s customers in its behavioral futures markets. They can 
rent a car without a deposit, receive favorable terms on loans and apartment rentals, receive fast-tracking 
for visa permits, enjoy being showcased on dating apps, and a host of other perks. However, one report 
warns that the privileges linked to a high personal credit score can suddenly tumble for reasons unrelated 
to consumption behavior, such as cheating on a university exam.“° 

In 2017 the central bank retracted its support for the private-sector personal credit programs, perhaps 
because they were too successful, their concentrations of knowledge and power too great. Sesame Credit 
had acquired more than 400 million users in just two years, staking a claim to just about every aspect of 
those users’ lives.2’ A journalist who wrote a book on Ant Financial anticipates that the government is 
preparing to assert control over the whole system: “The government doesn’t want this very important 
infrastructure of the people’s credit in one big company’s hands.” The Chinese government appears to 


understand that power accrues to the owners of the means of behavioral modification. It is the owners 
who set the behavioral parameters that define guaranteed outcomes. Thus, fusion advances. 

A sense of the kind of social world that might be produced by the fusion of instrumentarian and state 
power begins with the “judgment defaulter’s list,’ described by the Economist as the heart of the social 
credit system and perhaps the best indicator of its larger ambitions. The list includes debtors and anyone 
who has ever defied a court order: 


People on the list can be prevented from buying aeroplane, bullet-train or first- or business-class rail tickets; selling, buying or building 
a house; or enrolling their children in expensive fee-paying schools. There are restrictions on offenders joining or being promoted in 
the party and army, and on receiving honours and titles. If the defaulter is a company, it may not issue shares or bonds, accept foreign 


investment or work on government projects.22 


According to a report in China Daily, debtors on the list were automatically prevented from flying 
6.15 million times since the blacklist was launched in 2013. Those in contempt of court were denied sales 
of high-speed train tickets 2.22 million times. Some 71,000 defaulters have missed out on executive 
positions at enterprises as a result of their debts. The Industrial and Commercial Bank of China said it had 
refused loans worth more than 6.97 billion yuan ($1.01 billion) to debtors on the list.22 No one is sent to a 
reeducation camp, but they may not be allowed to purchase luxury goods. According to the director of the 
Institute of the Chinese Academy of International Trade and Economic Cooperation, “Given this 
inconvenience, 10 percent of people on the list started to pay back the money they owed spontaneously. 
This shows the system is starting to work.”42 Economies of action were performing to plan. 

For the 400 million users of Sesame Credit, the fusion of instrumentarian and state power bites hard. 
Those who might find themselves on the blacklist discover that the credit system is designed to thrust their 
scores into an inexorable downward spiral: “First your score drops. Then your friends hear you are on 
the blacklist and, fearful that their scores might be affected, quietly drop you as a contact. The algorithm 
notices, and your score plummets further.”4! 

The Chinese government’s vision may be impossibly ambitious: the big dream of total awareness and 
perfect certainty mediated by algorithms that filter a perpetual flood of data flows from private and public 
supplies, including online and offline experience culled from every domain and able to ricochet back into 
the individual lives of 1.5 billion people, automating social behavior as the algorithms reward, punish, 
and shape action right down to the latest bus ticket. So far the project is fragmented across many pilots, 
not only in the tech companies but also in cities and regions, so there is no real test of the scale that the 
government envisions. There are plenty of experts who believe that a single system of that scale and 
complexity will be difficult if not impossible to achieve. 

There are other good reasons to discount the social credit system’s relevance for our story. To state the 
obvious, China is not a democracy and its culture differs profoundly from Western culture. Syracuse 
University researcher Yang Wang observes that Chinese culture places less value on privacy than does 
Western culture and that most Chinese have accommodated to the certain knowledge of online government 
surveillance and censorship. The most common word for privacy, yinsi, didn’t even appear in popular 
Chinese dictionaries until the mid-1990s.“4 Chinese citizens have accepted national ID cards with 
biometric chips, “birth permits,” and now social credit rankings because their society has been saturated 
with surveillance and profiling for decades. For example, the “dang’an” is a wide-ranging personal 
dossier compiled on hundreds of millions of urban residents from childhood and maintained throughout 
life. This “Mao-era system for recording the most intimate details of life” is updated by teachers, 
Communist Party officials, and employers. Citizens have no rights to see its contents, let alone contest 


them. 

The dossier is only one feature of long-institutionalized and pervasive administrative systems of 
behavioral control and surveillance in daily life that bestow honors on some and punishments on others. 
Social control programs have expanded with the growth of the internet. For example, the “Golden Shield” 
is an extensive online surveillance system. China’s cyber-censors can suspend internet or social media 
accounts if their users send messages containing sensitive terms such as “Tibetan independence” or 
“Tiananmen Square incident.” 

As distinct as our politics and cultures may be or have been, the emerging evidence of the Chinese 
social credit initiatives broadcasts the logic of surveillance capitalism and the instrumentarian power that 
it produces. Sesame Credit doubles down on every aspect of surveillance capitalist operations, with 
hundreds of millions of people caught in the gears of an automated behavioral modification machine and 
its bubbling behavioral futures markets dispersing perks and honors like Pokémon fairy dust in return for 
guaranteed outcomes. 

Chinese users are rendered, classified, and queued up for prediction with every digital touch, and so 
are we. We are ranked on Uber, on eBay, on Facebook, and on many other web businesses, and those are 
only the rankings that we see. Chinese users are assigned a “character” score, whereas the US government 
urges the tech companies to train their algorithms for a “radicalism” score. Indeed, the work of the 
shadow text is to evaluate, categorize, and predict our behavior in millions of ways that we can neither 
know nor combat—these are our digital dossiers. When it comes to credit scoring, US and UK banks and 
financial services firms have floated business models based on the mining and analysis of social media 
data for credit scores. Facebook itself has hinted of its interest, even filing a patent.“ These efforts 
receded only because the Federal Trade Commission threatened regulatory intervention. 

Oxford University China scholar Rogier Creemers, who translated some of the first documents on the 
social credit system, observes that “the trend towards social engineering and ‘nudging’ individuals 
towards ‘better’ behavior is also part of the Silicon Valley approach that holds that human problems can 
be solved once and for all through the disruptive power of technology.... In that sense, perhaps the most 
shocking element of the story is not the Chinese government’s agenda, but how similar it is to the path 
technology is taking elsewhere.”“© 

In 2017 a surveillance technology trade show held in Shenzhen was packed with US companies selling 
their latest wares, especially cameras equipped with artificial intelligence and facial recognition. Among 
the crowd was the managing director of CCTV direct, a UK distributor of surveillance equipment. He 
lamented “how far behind the Western countries are,’ compared to the skills and thrills of China’s 
surveillance infrastructure, but he also comforted himself with this thought: “What starts here ends up in 
homes, airports, and businesses back in America.”“2 

The difference between surveillance capitalism in the West and China’s emerging social credit system 
pivots on the patterns of entanglement and engagement between instrumentarian and state power. There 
are structural differences. In the West, as we have seen, the patterns have taken on many forms. The state 
began as bosom and shelter, then eager student and envious cousin. Surveillance capitalism and its 
instruments have come of age now, producing a fitful but necessary partnership. Key instrumentarian 
capabilities are docked in the big surveillance capitalist firms, and the state must move with and through 
these firms to access much of the power it seeks. 

In the Chinese context, the state will run the show and own it, not as a market project but as a political 
one, a machine solution that shapes a new society of automated behavior for guaranteed political and 
social outcomes: certainty without terror. All the pipes from all the supply chains will carry behavioral 
surplus to this new, complex means of behavioral modification. The state will assume the role of the 


behaviorist god, owning the shadow text and determining the schedule of reinforcements and the 
behavioral routines that it will shape. Freedom will be forfeit to knowledge, but it will be the state’s 
knowledge that it exercises, not for the sake of revenue but for the sake of its own perpetuation. 


V. A Fork in the Road 


Recall Carl Friedrich’s observation on the challenge of grasping the naked facts of totalitarianism: 
“Virtually no one before 1914 anticipated the course of development which has overtaken Western 
civilization since then.... To this failure to foresee corresponds a difficulty in comprehending.” Recall 
too the grinning, robust “Joe” Stalin planted among Hollywood luminaries on the glossy pages of a 1939 
Look magazine. Will we suffer the same lack of foresight as those who could not comprehend 
totalitarianism’s rise, paralyzed by the sheer power of Big Other and its infinite echoes of consequence, 
distracted by our needs and confused by its speed, secrecy, and success? 

Astonishment is a necessary alarm. We need it, but it should not leave us frozen in disbelief. The 
steady drumbeat of Big Other’s manifest destiny, its breathtaking velocities, and the obscurity of its aims 
and purpose are intended to disarm, disorient, and bewilder. Inevitabilist ideology works to equate 
surveillance capitalism and its instrumentarian power with nature: not a human construction but something 
more like a river or a glacier, a thing that can only be joined or endured. All the more reason to ask: 
Might the banalities of today’s declarations (“instruct the vehicular monitoring system not to allow the car 
to be started”) also reveal themselves in the fullness of time as the seeds of our century’s greatest 
nightmare? What of the authors of the instrumentarian project? How will we appraise the smiling, robust 
faces of the tech titans when we revisit those images in the glossy pixels of some twenty-first-century 
version of Look? The road from Shenzhen to an American or European airport also leads to the Roomba 
vacuum cleaner mapping your living room and your breakfast with Alexa. It is the road to machine 
certainty imposed by instrumentarian power and produced by surveillance capitalism. This journey is not 
as long as you might think. 

There is a fork in the road. 

In one direction lies the possibility of a synthetic declaration for a third modernity based on the 
strengthening of democratic institutions and the creative construction of a double movement for our time. 
On this road we harness the digital to forms of information capitalism that reunite supply and demand in 
ways that are both genuinely productive of effective life and compatible with a flourishing democratic 
social order. The first step down this road begins with naming, establishing our bearings, reawakening 
our astonishment, and sharing a sense of righteous indignity. 

If we follow the other road, the one that links us to Shenzhen, we find our way to surveillance 
capitalism’s antidemocratic vision for a third modernity fashioned by instrumentarian power. It is a future 
of certainty accomplished without violence. The price we pay is not with our bodies but with our 
freedom. This future does not yet exist, but like Scrooge’s dream of Christmas future, the materials are all 
in place and ready for assembly. Chapter 14 examines this next way station on the road that began with an 
unprecedented capitalism, turned toward an unprecedented power, and now leads to an unprecedented 
society, theorized and legitimated by a burgeoning intellectual ecosystem of thinkers, researchers, and 
practitioners. What is this new place that they would have us call home? 
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Figure 4: Two Species of Power 


CHAPTER FOURTEEN 


A UTOPIA OF CERTAINTY 


So from the years their gifts were showered: each 
Grabbed at the one it needed to survive; 
Bee took the politics that suit a hive, 
Trout finned as trout, peach molded into peach, 
And were successful at their first endeavor. 


—W. H. AUDEN 
SONNETS FROM CHINA, I 


I. Society as the Other-One 


Although he did not name it, the visionary of ubiquitous computing, Mark Weiser, foresaw the immensity 
of instrumentarian power as a totalizing societal project. He did so in a way that suggests both its utter 
lack of precedent and the danger of confounding it with what has gone before: “hundreds of computers in 
every room, all capable of sensing people near them and linked by high-speed networks have the potential 
to make totalitarianism up to now seem like sheerest anarchy.”! In fact, all those computers are not the 
means to a digital hyper-totalitarianism. They are, as I think Weiser sensed, the foundation of an 
unprecedented power that can reshape society in unprecedented ways. If instrumentarian power can make 
totalitarianism look like anarchy, then what might it have in store for us? 

Seven decades ago, Skinner’s proto-instrumentarian behavioral utopia, Walden Two, was met with 
revulsion. Today the real thing is inspirational fodder for surveillance capitalist rhetoric as leaders 
promote the tools and visions that will bring the old professor’s ideas to life... to our lives. The 
processes of normalization and habituation have begun. We have already seen that surveillance 
capitalism’s pursuit of certainty—the mandate of the prediction imperative—requires a continuous 
approximation to total information as the ideal condition for machine intelligence. On the trail of totality, 
surveillance capitalists enlarged their scope from the virtual to the real world. The reality business 
renders all people, things, and processes as computational objects in an endless queue of equivalence 
without equality. Now, as the reality business intensifies, the pursuit of totality necessarily leads to the 
annexation of “society,” “social relations,” and key societal processes as a fresh terrain for rendition, 
calculation, modification, and prediction. 

Big Other’s ubiquity is revered as inevitable, but that is not the endgame. The aim in this new phase is 


the comprehensive visibility, coordination, confluence, control, and harmonization of social processes in 
the pursuit of scale, scope, and action. Although instrumentarianism and totalitarianism are distinct 
species, they each yearn toward totality, though in profoundly different ways. Totalitarianism seeks 
totality as a political condition and relies on violence to clear its path. Instrumentarianism seeks totality 
as a condition of market dominance, and it relies on its control over the division of learning in society, 
enabled and enforced by Big Other, to clear its path. The result is the application of instrumentarian 
power to societal optimization for the sake of market objectives: a utopia of certainty. 

Although they resonate in many respects with the instrumentarian social vision of China’s political 
elite, surveillance capitalists have distinct objectives. In their view, instrumentarian society is a market 
opportunity. Any norms and values they impose are designed to further the certain fulfillment of market 
goals. Like human experience, society is subordinated to the market dynamic and reborn as objectified 
computational behavioral metrics available to surveillance capitalism’s economies of scale, scope, and 
action in the pursuit of the most-lucrative supplies of behavioral surplus. In order to achieve these aims, 
surveillance capitalists have conjured a chilling vision. They aim to fashion a new society that emulates 
machine learning in much the same way that industrial society was patterned on the disciplines and 
methods of factory production. In their vision, instrumentarian power replaces social trust, Big Other 
substitutes certainty for social relations, and society as we know it shades into obsolescence. 


II. Totality Includes Society 


Like generals delivering a chest-thumping tally of their armies, surveillance capitalist leaders take care to 
assure allies of their great power. This is typically expressed in an inventory of the instrumentarian troops 
massed at the border, poised for the rendition of everything in pursuit of totality. This pursuit, it becomes 
clear, does not merely have consequences for society; it includes society. 

In the spring of 2017, Microsoft CEO Satya Nadella bounded onstage to open the company’s annual 
developers’ conference, his slender profile accentuated by the requisite black polo shirt, black jeans, and 
trendy black high-tops. He quickly dazzled the audience with a roll call of his troops. Nadella recounted 
the 500 million Windows 10 devices; 100 million monthly users of its Office software; 140 million 
monthly users of the corporation’s digital “assistant,” Cortana; and more than 12 million organizations 
signed on to its cloud services, including 90 percent of the Fortune 500. 

Nadella did not fail to remind his audience of the crushing velocity that drives the instrumentarian 
project in an explosion of shock and awe, especially in the years since surveillance capitalism came to 
dominate digital services: internet traffic increased by a factor of 17.5 million over 1992’s 100 gigabytes 
per day; 90 percent of the data in 2017 was generated in the prior two years; a single autonomous car will 
generate 100 gigabytes per second; there will be an estimated 25 billion intelligent devices by 2020. “It’s 
stunning to see the progress across the depth and breadth of our society and economy and how digital 
technology is so pervasive.... It’s about what you can do with that technology to have broad impact.” His 
final exhortation to the assembled developers—“Change the world!”—-earned a thunderous round of 
applause. 

In celebrating Google’s ambitions with the company’s developers in 2017, CEO Sundar Pichai ran 
parallel to Nadella, showcasing his troop strength as Google’s battalions fan out to embrace every corner 
of social life, demonstrating the breadth and depth of the corporation’s instrumentarian power with a zeal 
that would have made Professor Skinner glow. Pichai reports that seven of the company’s most-salient 
“products and platforms” engage one billion monthly active users, including Gmail, Android, Chrome, 


Maps, Search, YouTube, and the Google Play Store; two billion active Android devices; 800 million 
monthly active users of Google Drive with three billion objects uploaded each week; 500 million Photos 
users uploading 1.2 billion photos each day; 100 million devices using Google Assistant. Every device is 
recast as a vehicle for Assistant, which will be available “throughout the day, at home and on the go” for 
every kind of task or social function. Pichai wants even more, telling his team, “We must go deeper.” 
Assistant should be wherever “people might want to ask for help.” Google executives share the 
enthusiasm. “Technology is now on the cusp of taking us into a magical age,” writes Eric Schmidt, 
“solving problems today that we simply couldn’t solve on our own.”? Machine learning, he says, will do 
everything from curing blindness to saving animals from extinction. Above all, however, it is founder 
Larry Page who has long had his sights set on the transformation of society. 

“The societal goal is our primary goal,” Page told the Financial Times in 2016.4 “We need 
revolutionary change, not incremental change,” he told another interviewer that year. “We could probably 
solve a lot of the issues we have as humans.”? Much of Page’s future vision turns out to be stock utopian 
fare, themes that have been repeated for millennia. Page anticipates machine intelligence that restores 
humankind to the Garden of Eden, lifting us from toil and struggle into a new realm of leisure and 
fulfillment. He foresees, for example, a future society graced by “abundance” in all things, where 
employment is but a “crazy” distant memory.® 

Most unusual, however, is that Page portrays Google’s totalistic ambitions as a logical consequence of 
its commitment to the perfection of society. From his point of view, we should welcome the opportunity to 
lean on Big Other and willingly subordinate all knowledge and decision rights to Google’s plan. For the 
sake of the plan, the totality of society—every person, object, and process—must be corralled into the 
supply chains that feed the machines, which, in turn, spin the algorithms that animate Big Other to manage 
and mitigate our frailty: 


What you should want us to do is to really build amazing products and to really do that... we have to understand apps and we have to 
understand things you could buy, and we have to understand airline tickets. We have to understand anything you might search for. 
And people are a big thing you might search for.... We’re going to have people as a first class object in search... if we’re going to do 


a good job meeting your information needs, we actually need to understand things and we need to understand things pretty deeplyZ 


Total knowledge is sold as a requirement of the “preemptive” services that lead to the solution of 
solutions in the Al-powered, omniscient “Google Assistant”: 


It’s really trying to understand everything in the world and make sense of it.... A lot of queries are actually about places, so we need 
to understand places.... A lot of the queries are about content we can’t find. We did books, and so on.... So, we’ve been gradually 


expanding that... maybe you don’t want to ask a question. Maybe you want to just have it answered for you before you ask it. That 


would be better.2 


Google originated in the prospect of optimally organizing the world’s information, but Page wants the 
corporation to optimize the organization of society itself: “In my very long-term worldview,” he said in 
2013, “our software understands deeply what you’re knowledgeable about, what you’re not, and how to 
organize the world so that the world can solve important problems.”2 

Facebook CEO Mark Zuckerberg shares these totalistic ambitions, and he is increasingly frank about 
“society,” not just the individuals within it, as subordinate to Facebook’s embrace. His “three big 
company goals” include “connecting everyone; understanding the world; and building the knowledge 
economy, so that every user will have ‘more tools’ to share ‘different kinds of content.’”!2 Zuckerberg’s 


keen appreciation of second-modernity instabilities—and the yearning for support and connection that is 
among its most-vivid features—drives his confidence, just as it did for Google economist Hal Varian. The 
corporation would know every book, film, and song a person had ever consumed. Predictive models 
would enable the corporation to “tell you what bar to go to” when you arrive in a strange city. The vision 
is detailed: when you arrive at the bar, the bartender has your favorite drink waiting, and you’re able to 
look around the room and identify people just like you. 

Zuckerberg described the flow of behavioral surplus as “growing at an exponential rate... that lets us 
project into the future... two years from now people are going to be sharing twice as much... four years, 
eight times as much....” And in a nod to the already pressing competition for totality, Zuckerberg 
anticipated that Facebook’s social graph will “start to be a better map of how you navigate the web than 
the traditional link structure.” # 

To that end, the CEO told investors that Facebook would bring affordable internet access “to every 
person in the world” so that every user will have “more tools” to share “different kinds of content.”/4 
Nothing was likely to impede the corporation’s progress on the societal front, he asserted, because 
“humans have such a deep desire to express themselves.” 12 

In 2017 Zuckerberg went even further in articulating his societal ambitions, this time aiming straight at 
the heart of second-modernity anxieties: “People feel unsettled. A lot of what was settling in the past 
doesn’t exist anymore.” Zuckerberg believes that he and his company can provide a future “that works for 
everyone” and fulfills “personal, emotional, and spiritual needs” for “purpose and hope,” “moral 
validation,” and “comfort that we are not alone.” “Progress now requires humanity coming together not 
just as cities or nations,” Zuckerberg urged, “but also as a global community... the most important thing 
we at Facebook can do is develop the social infrastructure... to build a global community....” Citing 
Abraham Lincoln, Facebook’s founder located his company’s mission in the evolutionary time line of 
civilization, during which humanity organized itself first in tribes, then cities, then nations. The next phase 
of social evolution would be “global community,” and Facebook was to lead the way, constructing the 
means and overseeing the ends./4 

Speaking at Facebook’s 2017 developers’ conference, Zuckerberg linked his assertion of the 
company’s historic role in establishing a “global community” to the standard myth of the modern utopia, 
assuring his followers, “In the future, technology is going to... free us up to spend more time on the things 
we all care about, like enjoying and interacting with each other and expressing ourselves in new ways.... 
A lot more of us are gonna do what today is considered the arts, and that’s gonna form the basis of a lot of 
our communities. 

As Nadella and other surveillance capitalists spin their utopian dreams, the surveillance capitalists 
fail to mention that the magical age they envision comes at a price: Big Other must expand toward totality 
as it deletes all boundaries and overwhelms every source of friction in the service of its economic 
imperatives. All power yearns toward totality, and only authority stands in the way: democratic 
institutions; laws; regulations; rights and obligations; private governance rules and contracts; the normal 
market constraints exercised by consumers, competitors, and employees; civil society; the political 
authority of the people; and the moral authority of individual human beings who have their bearings. 

This point was made in Goethe’s fable of the sorcerer’s apprentice, when, in the absence of the 
sorcerer’s authority to guide and check the action, the apprentice transforms the broom into a demonic 
force of pure unrelenting power: 


Ah, the word with which the master 
Makes the broom a broom once more! 


Ah, he runs and fetches faster! 
Be a broomstick as before! 
Ever new the torrents 

That by him are fed, 

Ah, a hundred currents 

Pour upon my head!48 


III. Applied Utopistics 


Instrumentarian power, like the apprentice’s broom, has flourished in the sorcerer’s absence with little 
authority to check its action, and the surveillance capitalists’ appetite for totality has grown with this 
success. The utopian rhetoric of a magical age has been critical to this progress. The notion that Big Other 
will solve all of humanity’s problems while empowering each individual is usually dismissed as mere 
“techno utopianism,” but it would be a mistake for us to ignore this rhetoric without examining its 
purpose. Such discourse is no mere hogwash. It is the minesweeper that precedes the foot soldiers and the 
canny diplomat sent ahead to disarm the enemy and smooth the way for a quiet surrender. The promise of 
a magical age plays a critical strategic role, simultaneously distracting us from and legitimating 
surveillance capitalism’s totalistic ambitions that necessarily include “people” as a “first class object.” 

The “societal goal” articulated by the leading surveillance capitalists fits snugly into the notion of 
limitless technological progress that dominated utopian thought from the late eighteenth century through 
the late nineteenth century, culminating with Marx. Indeed, surveillance capitalists such as Nadella, Page, 
and Zuckerberg conform to five of the six elements with which the great scholars of utopian thought, Frank 
and Fritzie Manuel, define the classic profile of the most ambitious modern utopianists: (1) a tendency 
toward highly focused tunnel vision that simplifies the utopian challenge, (2) an earlier and more 
trenchant grasp of a “new state of being” than other contemporaries, (3) the obsessive pursuit and defense 
of an idée fixe, (4) an unshakable belief in the inevitability of one’s ideas coming to fruition, and (5) the 
drive for total reformation at the level of the species and the entire world system. 

The Manuels observe a sixth characteristic of the future-oriented modern visionary, and this is where 
the men and the corporations examined here represent powerful exceptions to the rule: “Often a utopian 
foresees the later evolution and consequences of technological development already present in an 
embryonic state; he may have antennae sensitive to the future. His gadgets, however, rarely go beyond the 
mechanical potentialities of his age. Try as he may to invent something wholly new, he cannot make a 
world out of nothing.”+2 In our time, however, surveillance capitalists can and do make such a world—a 
genuinely historic deviation from the norm. 

Individually and collectively, the knowledge, power, and wealth that surveillance capitalists command 
would be the envy of any ancient potentate, just as they are now coveted by the modern state. With 2017 
balance sheets reporting $126 billion in cash and securities for Microsoft, $92 billion for Google, and 
about $30 billion for Facebook, and the financial markets endorsing their ever-expanding instrumentarian 
regimes with more than $1.6 trillion in market capitalization in mid-2017, these are the rare utopianists 
who can oversee the translation of their imaginations into fact without soldiers to pave the way in 
blood./2 

In this respect, the surveillance capitalist leaders are sui generis utopianists. Marx grasped the world 
with his thickly articulated theory, but with only the power of his ideas, he could not implement his vision. 
Long after the publication of Marx’s theories, men such as Lenin, Stalin, and Mao applied them to real 


life. Indeed, the Manuels describe Lenin as a specialist in “applied utopistics.”22 In contrast, the 
surveillance capitalists seize the world in practice. Their theories are thin—at least this is true of the 
thinking that they share with the public. The opposite is true of their power, which is monumental and 
largely unimpeded. 

When it comes to theory and practice, the usual sequence is that theory is available to inspect, 
interrogate, and debate before action is initiated. This allows observers an opportunity to judge a theory’s 
worthiness for application, to consider unanticipated consequences of application, and to evaluate an 
application’s fidelity to the theory in which it originates. The unavoidable gap between theory and 
practice creates a space for critical inquiry. For example, we can question whether a law or governmental 
practice is consistent with a nation’s constitution, charter of rights, and governing principles because we 
can inspect, interpret, and debate those documents. If the gap is too great, citizens act to close the gap by 
challenging the law or practice. 

The surveillance capitalists reverse the normal sequence of theory and practice. Their practices move 
ahead at high velocity in the absence of an explicit and contestable theory. They specialize in displays of 
instrumentarianism’s unique brand of shock and awe, leaving onlookers dazed, uncertain, and helpless. 
The absence of a clear articulation of their theory leaves the rest of us to ponder its practical effects: the 
vehicular monitoring system that shuts down your engine; the destination that appears with the route; the 
suggested purchase that flashes on your phone the moment your endorphins peak; Big Other’s continuous 
tracking of your location, behavior, and mood; and its cheerful herding of city dwellers toward 
surveillance capitalism’s customers. 

However meager and secretive the surveillance capitalists’ theories might be, the instrumentarian 
power they wield can make their dreams come true, or at least inflict a whirlwind of consequences as 
they try. The only way to grasp the theory advanced in their applied utopistics is to reverse engineer their 
operations and scrutinize their meaning, as we have done throughout these chapters. 

Applied utopistics are on the move at Facebook, Google, and Microsoft as the frontier of behavioral 
surplus extraction moves into realms of life traditionally understood as societal and elaborated under 
some combination of civil institutions and public leadership. Zuckerberg’s 2017 mission statement for 
Facebook, introduced as “building global community,” announced a new phase of applied utopistics: 
“Overall, it is important that the governance of our community scales with the complexity and demands of 
its people. We are committed to always do better, even if that involves building a worldwide voting 
system to give you more voice and control. Our hope is that this model provides examples of how 
collective decision-making may work in other aspects of the global community.”=! Later that year, 
Zuckerberg told an audience of developers that “we have a full roadmap of products to help build groups 
and community, help build a more informed society, help keep our communities safe, and we have a lot 
more to do here.” 24 

Back on that stage in the spring of 2017, Microsoft’s Nadella encouraged his developers: “Whether 
it’s precision medicine or precision agriculture, whether it’s digital media or the industrial internet, the 
opportunity for us as developers to have broad, deep impact on all parts of society and all parts of the 
economy has never been greater.”22 The vision that Nadella unveiled that day is emblematic of the wider 
surveillance capitalist template for our futures. Where do they think they are taking us? 


IV. Confluence as Machine Relations 


In order to decipher the true measure of an instrumentarian society, let’s set aside the razzle-dazzle of a 


“magic age” and focus instead on the practices of applied utopistics and the social vision they imply. 
Nadella provided a valuable opportunity when he unveiled a series of practical applications that imply a 
sweeping new vision of machine relations as the template for a new era’s social relations. 

The reveal begins with Nadella’s account of a Microsoft collaboration with a 150-year-old Swedish 
manufacturer of high-precision metal-cutting equipment that has reinvented itself for the twenty-first 
century. The project is a state-of-the-art illustration of what Nadella describes as the “fundamental change 
in the paradigm of the apps that we are building, a change in the worldview that we have... from... a 
mobile-first, cloud-first world to a new world that is going to be made up of an intelligent cloud and an 
intelligent edge.” Artificial intelligence, he says, “learns from information and interacts with the physical 
world,” thus citing the capabilities required for economies of action.24 

Nadella first describes the machines linked by telemetry in the new factory setting as they continuously 
stream data to the “IoT hub” in the “cloud,” where Microsoft’s analyses search out anomalies that could 
put the machines at risk. Each anomaly is traced back through the data stream to its cause, and machine 
intelligence in the hub learns to identify the causal patterns so that it can preemptively shut down a 
threatened piece of equipment in about two seconds, before a potentially damaging event can occur. 

Then Nadella describes the new “breakthrough capability” in which a computational actuating sensor 
is embedded directly in the machine, dramatically reducing the time to a preemptive shutdown: “That 
logic is now running locally, so there’s no cloud loop.” The “edge” knows immediately when the machine 
experiences an event that predicts a future anomaly, and it shuts down the equipment within 100 
milliseconds of this computation, a “20X improvement.” This is celebrated as “the power of the cloud 
working in harmony with an intelligent edge” to anticipate and preempt variations from the norm 
“before they happen.”22 

The power of machine learning develops exponentially as the devices learn from one another’s 
experiences, feeding into and drawing upon the intelligence of the hub. In this scenario it’s not that the 
whole is greater than the sum of the parts; it’s more like there are no parts. The whole is everywhere, fully 
manifest in each device embedded in each machine. Nadella translates these facts into their practical 
application, observing that once you have lots of devices around, an “ad hoc data center” is created “ona 
factory floor, at home, or anywhere else.... You can turn any place into a safe, Al-driven place.”© 

With this statement it finally becomes clear that “safe” means “automatically anomaly-free.” In 
Nadella’s factory, machine knowledge instantaneously replaces ignorance, herding all machine behavior 
to preestablished norms. Rather than concern for the multiplication of risk and the contagion of failure 
should machine learning go awry, Nadella celebrates the synchrony and universality of certain outcomes, 
as every machine is the same machine marching to the same song. 

Just as a century ago the logic of mass production and its top-down administration provided the 
template for the principles of industrial society and its wider civilizational milieu, so too is Nadella’s 
new-age factory revealed as the proving ground for his social vision—surveillance capitalism’s vision— 
of an instrumentarian society enabled by a new form of collective action. Machine learning is rendered 
here as a collective mind—a hive mind—in which each element learns and operates in concert with every 
other element, a model of collective action in which all the machines in a networked system move 
seamlessly toward confluence, all sharing the same understanding and operating in unison with maximum 
efficiency to achieve the same outcomes. Confluent action means that the “freedom” of each individual 
machine is surrendered to the knowledge that they share. Just as the behavioral theorists Planck, Meyer, 
and Skinner predicted, this sacrifice amounts to an all-out war on accidents, mistakes, and randomness in 
general. 

Nadella takes this template of new machine relations and applies it to a more complex illustration of a 


human and machine system, though still in the “economic domain.” This time it’s a construction site, 
where human and machine behaviors are tuned to preestablished parameters determined by superiors and 
referred to as “policies.” Algorithmic uncontracts apply rules and substitute for social functions such as 
supervision, negotiation, communication, and problem solving. Each person and piece of equipment takes 
a place among an equivalence of objects, each one “recognizable” to “the system” through the AI devices 
distributed across the site. 

For example, each individual’s training, credentials, employment history, and other background 
information are instantly on display to the system. A “policy” might declare that “only credentialed 
employees can use jackhammers.” If an employee who is not accredited for jackhammer use approaches 
that tool, the possibility of an impending violation is triggered, and the jackhammer screams an alert, 
instantly disabling itself. 

Significantly, it is not only the unified action of things on the site that are mobilized in alignment with 
policies. Confluent human action is also mobilized, as social influence processes are triggered in the 
preemptive work of anomaly avoidance. In the case of the at-risk jackhammer, the humans at the site are 
mobilized to swarm toward the location of the Al-anticipated anomalous offense in order to “quickly 
resolve it.” “The intelligent edge,” Microsoft developers are told, “is the interface between the computer 
and the real world... you can search the real world for people, objects and activities, and apply policies 
to them....”22 

Once people and their relationships are rendered as otherized, equivalent “things in the cloud,” 25 
billion computational actuating devices can be mobilized to shape behavior around safe and harmonious 
“policy” parameters. The most “profound shift,” Nadella explained, is that “people and their relationship 
with other people is now a first-class thing in the cloud. It’s not just people but it’s their relationships, 
it’s their relationships to all of the work artifacts, their schedules, their project plans, their documents; all 
of that now is manifest in this Microsoft Graph.” These streams of total information are key to optimizing 
“the future of productivity,” Nadella exulted.72 

In Microsoft’s instrumentarian society, the factories and workplaces are like Skinner’s labs, and the 
machines replace his pigeons and rats. These are the settings where the architecture and velocities of 
instrumentarian power are readied for translation to society in a digital-age iteration of Walden Two in 
which machine relations are the model for social relations. Nadella’s construction site exemplifies the 
grand confluence in which machines and humans are united as objects in the cloud, all instrumented and 
orchestrated in accordance with the “policies.” The magnificence of “policies” lies precisely in the fact 
that they appear on the scene as guaranteed outcomes to be automatically imposed, monitored, and 
maintained by the “system.” They are baked into Big Other’s operations, an infinity of uncontracts 
detached from any of the social processes associated with private or public governance: conflict and 
negotiation, promise and compromise, agreement and shared values, democratic contest, legitimation, and 
authority. 

The result is that “policies” are functionally equivalent to plans, as Big Other directs human and 
machine action. It ensures that doors will be locked or unlocked, car engines will shut down or come to 
life, the jackhammer will scream “no” in suicidal self-sacrifice, the worker will adhere to norms, the 
group will swarm to defeat anomalies. We will all be safe as each organism hums in harmony with every 
other organism, less a society than a population that ebbs and flows in perfect frictionless confluence, 
shaped by the means of behavioral modification that elude our awareness and thus can neither be mourned 
nor resisted. 

Just as the division of labor migrated from the economic domain to society in the twentieth century, 
Nadella’s construction site is the economic petri dish in which a new division of learning mutates into 


life, ready for translation to society. In the twentieth century the critical success factors of industrial 
capitalism—efficiency, productivity, standardization, interchangeability, the minute division of labor, 
discipline, attention, scheduling, conformity, hierarchical administration, the separation of knowing and 
doing, and so forth—were discovered and crafted in the workplace and then transposed to society, where 
they were institutionalized in schools, hospitals, family life, and personality. As generations of scholars 
have documented, society became more factory-like so that we might train and socialize the youngest 
among us to fit the new requirements of a mass production order. 

We have entered this cycle anew, but now the aim is to remake twenty-first-century society as a “first- 
class thing” organized in the image of the machine hive for the sake of others’ certainty. The 
connectedness that we once sought for personal sustenance and effectiveness is recast as the medium for a 
new species of power and the social confluence that translates into guaranteed outcomes. 


V. Confluence as Society 


Microsoft scientists have been working for years on how to take the same logic of automated preemptive 
control at the network’s edge and transpose it to social relations. As Nadella observed in 2017, if “we” 
can do this in a “physical place,” it can be done “everywhere” and “anywhere.” He advised his audience 
of applied utopianists, “You could start reasoning about people, their relationship with other people, the 
things in the place....”22 

The imaginative range of this new thinking is demonstrated in a 2013 Microsoft patent application 
updated and republished in 2016 and titled “User Behavior Monitoring on a Computerized Device.”=2 
With conspicuously thin theory complemented by thick practice, the patented device is designed to 
monitor user behavior in order to preemptively detect “any deviation from normal or acceptable behavior 
that is likely to affect the user’s mental state. A prediction model corresponding to features of one or more 
mental states may be compared with features based upon current user behavior.” 

The scientists propose an application that can sit in an operating system, server, browser, phone, or 
wearable device continuously monitoring a person’s behavioral data: interactions with other people or 
computers, social media posts, search queries, and online activities. The app may activate sensors to 
record voice and speech, videos and images, and movement, such as detecting “when the user engages in 
excessive shouting by examining the user’s phone calls and comparing related features with the 
predication model.” 

All these behavioral data are stored for future historical analyses in order to improve the prediction 
model. If the user normally restrains the volume of his or her voice, then sudden excessive shouting may 
indicate a “psychosocial event.” Alternatively, the behavior could be assessed in relation to a “feature 
distribution representing normal and/or acceptable behavior for an average member of a population... a 
Statistically significant deviation from that behavior baseline indicates a number of possible 
psychological events.” The initial proposition is that in the event of an anomaly, the device would alert 
“trusted individuals” such as family members, doctors, and caregivers. But the circle widens as the patent 
specifications unfold. The scientists note the utility of alerts for health care providers, insurance 
companies, and law-enforcement personnel. Here is a new surveillance-as-a-service opportunity geared 
to preempt whatever behavior clients choose. 

Microsoft’s patent returns us to Planck, Meyer, and Skinner and the viewpoint of the Other-One. In 
their physics-based representation of human behavior, anomalies are the “accidents” that are called 
freedom but actually denote ignorance; they simply cannot yet be explained by the facts. 


Planck/Meyer/Skinner believed that the forfeit of this freedom was the necessary price to be paid for the 
“safety” and “harmony” of an anomaly-free society in which all processes are optimized for the greater 
good. Skinner imagined that with the correct technology of behavior, knowledge could preemptively 
eliminate anomalies, driving all behavior toward preestablished parameters that align with social norms 
and objectives. “If we could show that our members preferred life in Walden Two,” says Frazier-Skinner, 
“it would be the best possible evidence that we had reached a safe and productive social structure.” 

In this template of social relations, behavioral modification operates just beyond the threshold of 
human awareness to induce, reward, goad, punish, and reinforce behavior consistent with “correct 
policies.” Thus, Facebook learns that it can predictably move the societal dial on voting patterns, 
emotional states, or anything else that it chooses. Niantic Labs and Google learn that they can predictably 
enrich McDonald’s bottom line or that of any other customer. In each case, corporate objectives define the 
“policies” toward which confluent behavior harmoniously streams. 

The machine hive—the confluent mind created by machine learning—is the material means to the final 
elimination of the chaotic elements that interfere with guaranteed outcomes. Eric Schmidt and Sebastian 
Thrun, the machine intelligence guru who once directed Google’s X Lab and helped lead the development 
of Street View and Google’s self-driving car, make this point in championing Alphabet’s autonomous 
vehicles. “Let’s stop freaking out about artificial intelligence,” they write. 

Schmidt and Thrun emphasize the “crucial insight that differentiates AI from the way people learn. 
Instead of the typical assurances that machines can be designed to be more like human beings and 
therefore less threatening, Schmidt and Thrun argue just the opposite: it is necessary for people to become 
more machine-like. Machine intelligence is enthroned as the apotheosis of collective action in which all 
the machines in a networked system move seamlessly toward confluence, all sharing the same 
understanding and thus operating in unison with maximum efficiency to achieve the same outcomes. The 
jackhammers do not independently appraise their situation; they each learn what they all learn. They each 
respond the same way to uncredentialed hands, their brains operating as one in service to the “policy.” 
The machines stand or fall together, right or wrong together. As Schmidt and Thrun lament, 
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When driving, people mostly learn from their own mistakes, but they rarely learn from the mistakes of others. People collectively 
make the same mistakes over and over again. As a result, hundreds of thousands of people die worldwide every year in traffic 
collisions. AI evolves differently. When one of the self-driving cars makes an error, all of the self-driving cars learn from it. In fact, 
new self-driving cars are “born” with the complete skill set of their ancestors and peers. So collectively, these cars can learn faster 
than people. With this insight, in a short time self-driving cars safely blended onto our roads alongside human drivers, as they kept 


learning from each other’s mistakes.... Sophisticated AlI-powered tools will empower us to better learn from the experiences of 


others.... The lesson with self-driving cars is that we can learn and do more collectively.23 


This is a succinct but extraordinary statement of the machine template for the social relations of an 
instrumentarian society. The essence of these facts is that first, machines are not individuals, and second, 
we should be more like machines. The machines mimic each other, and so must we. The machines move 
in confluence, not many rivers but one, and so must we. The machines are each structured by the same 
reasoning and flowing toward the same objective, and so must we be structured. 

The instrumentarian future integrates this symbiotic vision in which the machine world and social 
world operate in harmony within in and across “species” as humans emulate the superior learning 
processes of the smart machines. This emulation is not intended as a throwback to mass production’s 
Taylorism or Chaplin’s hapless worker swallowed by the mechanical order. Instead, this prescription for 
symbiosis takes a different road on which human interaction mirrors the relations of the smart machines as 


individuals learn to think and act by emulating one another, just like the self-driving cars and the policy- 
worshipping jackhammers. 

In this way, the machine hive becomes the role model for a new human hive in which we march in 
peaceful unison toward the same direction based on the same “correct” understanding in order to 
construct a world free of mistakes, accidents, and random messes. In this world the “correct” outcomes 
are known in advance and guaranteed in action. The same ubiquitous instrumentation and transparency that 
define the machine system must also define the social system, which in the end is simply another way of 
describing the ground truth of instrumentarian society. 

In this human hive, individual freedom is forfeit to collective knowledge and action. Nonharmonious 
elements are preemptively targeted with high doses of tuning, herding, and conditioning, including the full 
seductive force of social persuasion and influence. We march in certainty, like the smart machines. We 
learn to sacrifice our freedom to collective knowledge imposed by others and for the sake of their 
guaranteed outcomes. This is the signature of the third modernity offered up by surveillance capital as its 
answer to our quest for effective life together. 


CHAPTER FIFTEEN 


THE INSTRUMENTARIAN COLLECTIVE 


So an age ended, and its last deliverer died 
In bed, grown idle and unhappy; they were safe: 
The sudden shadow of a giant’s enormous calf 
Would fall no more at dusk across their lawns outside. 


—W. H. AUDEN 
SONNETS FROM CHINA, X 


I. The Priests of Instrumentarian Power 


Applied utopianist executives such as Page, Nadella, and Zuckerberg do not say much about their 
theories. At best the information we have is episodic and shallow. But a cadre of data scientists and 
“computational social scientists” has leapt into this void with detailed experimental and theoretical 
accounts of the gathering momentum of instrumentarian power, providing invaluable insight into the social 
principles of an instrumentarian society. 

One outstanding example is the work of Alex Pentland, the director of the Human Dynamics Lab 
within MIT’s Media Lab. Pentland is the rare applied utopianist who, in collaboration with his students 
and collaborators, has vigorously articulated, researched, and disseminated a theory of instrumentarian 
society in parallel to his prolific technical innovations and practical applications. The studies that this 
group has produced are a contemporary signal of an increasingly taken-for-granted worldview among 
data scientists whose computational theories and innovations exist in dynamic interaction with the 
progress of surveillance capitalism, as in the case of Picard’s affective computing and Paradiso’s digital 
omniscience. However, few consider the social ramifications of their work with Pentland’s insight and 
conviction, providing us with an invaluable opportunity to critically explore the governance assumptions, 
societal principles, and social processes that define an instrumentarian society. My aim is to infer the 
theory behind the practice, as surveillance capitalists integrate “society” as a “first class object” for 
rendition, computation, modification, monetization, and control. 

Pentland is a prolific author or coauthor of hundreds of articles and research studies in the field of 
data science and is a prominent institutional actor who advises a roster of organizations, including the 
World Economic Forum, the Data-Pop Alliance, Google, Nissan, Telefonica, and the Office of the United 
Nations Secretary General. Pentland’s research lab is funded by a who’s who of global corporations, 


consultancies, and governments: Google, Cisco, IBM, Deloitte, Twitter, Verizon, the EU Commission, the 
US government, the Chinese government, “and various entities who are all concerned with why we don’t 
know what’s going on in the world....”4 

Although Pentland is not alone in this field, he is something of a high priest among an exclusive group 
of priests. Unlike Hal Varian, Pentland does not speak of Google in the first-person plural, but his work is 
showcased in surveillance capitalist enclaves, where it provides the kind of material and intellectual 
support that helps to legitimate instrumentarian practices. Appearing for a presentation at Google, where 
Pentland is on the Advisory Board for the Advanced Technology and Projects Group, former Pentland 
doctoral student and top Google executive Brad Horowitz introduced his mentor as an “inspirational 
educator” with credentials across many disciplines and whose former students lead the computational 
sciences in theory and practice.2 

Pentland is often referred to as the “godfather of wearables,” especially Google Glass. In 1998 he 
predicted that wearables “can extend one’s senses, improve memory, aid the wearer’s social life and 
even help him or her stay calm and collected.”? Thad Starner, one of Pentland’s doctoral students, 
developed a primitive “wearable” device while at MIT and was hired by Sergey Brin in 2010 to continue 
that work at Google: a project that produced Google Glass. More than fifty of Pentland’s doctoral students 
have gone on to spread the instrumentarian vision in top universities, in industry research groups, and in 
thirty companies in which Pentland participates as cofounder, sponsor, or advisor. Each one applies some 
facet of Pentland’s theory, analytics, and inventions to real people in organizations and cities.4 

Pentland’s academic credentials and voluble intelligence help legitimate a social vision that repelled 
and alarmed intellectuals, public officials, and the general public just decades ago. Most noteworthy is 
that Pentland “completes” Skinner, fulfilling his social vision with big data, ubiquitous digital 
instrumentation, advanced mathematics, sweeping theory, numerous esteemed coauthors, institutional 
legitimacy, lavish funding, and corporate friends in high places without having attracted the worldwide 
backlash, moral revulsion, and naked vitriol once heaped on Harvard’s outspoken behaviorist. This fact 
alone suggests the depth of psychic numbing to which we have succumbed and the loss of our collective 
bearings. 

Like Skinner, Pentland is a designer of utopias and a lofty thinker quick to generalize from animals to 
the entire arc of humanity. He is also a hands-on architect of instrumentarianism’s practical architecture 
and computational challenges. Pentland refers to his theory of society as “social physics,” a conception 
that confirms him as this century’s B. F. Skinner, by way of Planck, Meyer, and MacKay.? And although 
Pentland never mentions the old behaviorist, Pentland’s book, Social Physics, summons Skinner’s social 
vision into the twenty-first century, now fulfilled by the instruments that eluded Skinner in his lifetime. 
Pentland validates the instrumentarian impulse with research and theory that are boldly grounded in 
Skinner’s moral reasoning and epistemology as captured by the viewpoint of the Other-One. 

Professor Pentland began his intellectual journey as Skinner did, in the study of animal behavior. 
Where Skinner trained his reasoning on the detailed behaviors of blameless individual creatures, Pentland 
concerned himself with the gross behavior of animal populations. As a part-time researcher at NASA’s 
Environmental Research Institute while still an undergraduate, he developed a method for assessing the 
Canadian beaver population from space by counting the number of beaver ponds: “You’re watching the 
lifestyle, and you get an indirect measure.”® 

The experience appears to have hooked Pentland on the distant detached gaze, which he would later 
embrace as the “God’s eye view.” You may have experienced the sensation of the God view from the 
window Seat of an airplane as it lifts you above the city, transforming all the joys and woes below into the 
mute bustle of an anthill. Up there, any sense of “we” quickly dissolves into the viewpoint of the Other- 


One, and it is this angle of observation that founded Pentland’s science as he learned to apply MacKay’s 
principles of remote observation and telestimulation to humans: “If you think about people across the 
room talking, you can tell a lot.... It’s like watching beavers from outer space, like Jane Goodall watching 
gorillas. You observe froma distance.” (This is a slur on Goodall, of course, whose seminal genius was 
her ability to understand the gorillas she studied not as “other ones” but rather as “one of us.”) 

The God view would come to be essential to the conception of instrumentarian society, but a 
comprehensive picture emerged gradually over years of piecemeal experimentation. In the following 
section we track that journey as Pentland and his students learned to render, measure, and compute social 
behavior. With that foundation, we turn to Pentland’s Social Physics, which aims to recast society as an 
instrumentarian hive mind—like Nadella’s machines—but now extensively theorized and deeply 
evocative of Skinner’s formulations, values, worldview, and vision of the human future. 


IT. When Big Other Eats Society: The Rendition of Social Relations 


Skinner bitterly lamented the absence of “instruments and methods” for the study of human behavior 
comparable to those available to physicists. As if in response, Pentland and his students have spent the 
last two decades determined to invent the instruments and methods that can transform all of human 
behavior, especially social behavior, into highly predictive math. An early milestone was a 2002 
collaboration with then-doctoral student Tanzeem Choudhury, in which the coauthors wrote, “As far as we 
know, there are currently no available methods to automatically model face-to-face interactions. This 
absence is probably due to the difficulty of obtaining reliable measurements from real-world interactions 
within a community.... We believe sensing and modeling physical interactions among people is an 
untapped resource.”® In other words, the “social” remained an elusive domain even as data and computers 
had become more commonplace. 

The researchers’ response was to introduce the “sociometer,” a wearable sensor that combines a 
microphone, accelerometer, Bluetooth connection, analytic software, and machine learning techniques 
designed to infer “the structure and dynamic relationships” in human groups.2 (Choudhury would 
eventually run the People Aware Computing group at Cornell University.) From that point onward, 
Pentland and his teams have worked to crack the code on the instrumentation and instrumentalization of 
social processes in the name of a totalistic social vision founded on a comprehensive means of behavior 
modification. 

A 2005 collaboration with doctoral student Nathan Eagle reiterated the problem of insufficient data on 
human society, noting the “bias, sparsity of data, and lack of continuity” in social science’s understanding 
of human behavior and the resulting “absence of dense continuous data that also hinders the machine 
learning and agent-based modeling communities from constructing more comprehensive predictive 
models of human dynamics.”!2 Pentland had insisted that even the relatively new field of “data mining” 
could not capture the “real action” of conversations and face-to-face interactions necessary for a trenchant 
and comprehensive grasp of social behavior. But he also recognized that a rapidly growing swath of 
human activity—from transactions to communication—was falling to computer mediation, largely as a 
result of the cell phone. 

The team saw that it would be possible to exploit the increasingly “ubiquitous infrastructure” of 
mobile phones and combine those data with new streams of information from their wearable behavioral 
monitors. The result was a radical new solution that Pentland and Eagle called “reality mining.” Mentor 
and student demonstrated how the data from cell phones “can be used to uncover regular rules and 


structure in the behavior of both individuals and organizations,” thus furthering the progress of behavioral 
surplus capture and analysis and pointing the way toward the larger shift in the nature of behavioral 
dispossession from virtual, to actual, to social experience. As a technological and cultural landmark, the 
researchers’ announcement that “reality” was now fair and feasible game for surplus capture, search, 
extraction, rendition, datafication, analysis, prediction, and intervention helped to forge a path toward the 
new practices that would eventually become the “reality business.” 

Pentland and Eagle began with 100 MIT students and faculty centered at the Media Lab, equipping 
them with 100 Nokia phones preloaded with special software in a project that would become the basis of 
Eagle’s doctoral dissertation. The two researchers exposed the revelatory power of continuously 
harvested behavioral data, which they validated with survey information collected directly from each 
participant. Their analyses produced detailed portraits of individual and group life: the “social system,” 
as the authors called it. They were able to specify regular temporal and spatial patterns of location, 
activity, and communication-use patterns, which together enabled predictions of up to 90 percent accuracy 
regarding where someone was likely to be and what that person was likely to be doing within the hour, as 
well as highly accurate predictions about an individual’s colleagues, casual friends, and close 
relationships. The team identified patterns of communication and interaction within work groups, as well 
as the broad “organizational rhythms and network dynamics” of the Media Lab. (Eagle became CEO of 
Jana, a mobile advertising company that offers free internet to emerging markets in exchange for 
behavioral surplus.) 

As the theory and practice of reality mining continued to evolve in Pentland’s lab, work projects, and 
theories, the MIT Technology Review singled out “reality mining” as one of its “10 Breakthrough 
Technologies” in 2008. “My students and I have created two behavior-measurement platforms to speed 
the development of this new science,” Pentland said. “These platforms today produce vast amounts of 
quantitative data for hundreds of research groups around the world.”/2 

This allegiance to speed is, as we know, not a casual description but rather a key element in the art 
and science of applied utopistics. Pentland understands the rapid encroachments of Big Other and 
instrumentarian power as a “light-speed, hyperconnected world” where virtual crowds of millions from 
anywhere in the world “can form in minutes.” He views the MIT community as the avant-garde: brilliant 
pioneers of light speed, already in sync with its extreme velocities and thus a model for the rest of society. 
Reflecting on his students and colleagues, Pentland writes that “I have also gotten to see how creative 
cultures must change in order to thrive in the hyperconnected, warp-speed world that is MIT, an 
environment that the rest of the world is now entering.”44 Pentland reasons that his group’s adaptation to 
MIT’s norms of rapid deployment simply foreshadows what is in store for the rest of us. 

In the MIT Technology Review’s enthusiastic 2008 tribute to “reality mining,” it noted the then-still- 
new and disturbing facts of behavioral surplus: “Some people are nervous about trailing digital bread 
crumbs behind them. Sandy Pentland however revels in it.” Pentland would like to see phones collect 
“even more information” about their users: “It’s an interesting God’s-eye view.” Indeed, Pentland 
regularly celebrates “the predictive power of digital breadcrumbs” in his articles, indulging in the 
euphemisms and thin rationalizations that are also standard fare for surveillance capitalists and that 
contribute to the normalization of the dispossession of human experience. He says, for example, 


As we go about our daily lives, we leave behind virtual breadcrumbs—digital records of the people we call, the places we go, the 


things we eat and the products we buy. These breadcrumbs tell a more accurate story of our lives than anything we choose to reveal 


about ourselves.... Digital breadcrumbs... record our behavior as it actually happened.2& 


Pentland was among the first to recognize the commercial relevance of behavioral surplus. Although 
he does not discuss it explicitly, he appears to embrace the realpolitik of surveillance capitalism as the 
necessary condition for an instrumentarian society. Pentland’s own companies are extensions of his 
applied utopistics: proving grounds for instrumentarian techniques and the habituation of populations to 
pervasive rendition, monitoring, and modification in pursuit of surveillance revenues. 

From the start, Pentland understood reality mining as the gateway to a new universe of commercial 
opportunities. In 2004 he asserted that cell phones and other wearable devices with “computational 
horsepower” would provide the “foundation” for reality mining as an “exciting new suite of business 
applications.” The idea was always that businesses could use their privileged grasp of “reality” to shape 
behavior toward maximizing business objectives. He describes new experimental work in which speech- 
recognition technology generated “profiles of individuals based on the words they use,” thus enabling a 
manager to “forma team of employees with harmonious social behavior and skills.” 

In their 2006 article, Pentland and Eagle explained that their data would be “of significant value in the 
workplace,” and the two jointly submitted a patent for a “combined short range radio network and 
cellular telephone network for interpersonal communications” that would add to the stock of instruments 
available for businesses to mine reality.12 Eagle told Wired that year that the reality mining study 
represented an “unprecedented data set about continuous human behavior” that would revolutionize the 
study of groups and offer new commercial applications. He was reported to be “in talks” with a large 
company that already wanted to apply his instruments and methods./2 Pentland argued that information 
gathered by his sociometers—“unobtrusive wearable sensors” measuring communication, voice tones, 
and body language—‘“could help managers understand who is working with whom and infer the 
relationships between colleagues” and “would be an efficient way to find people who might work well 
together.” 22 

In a 2009 collaboration with several graduate students, Pentland presented results on the design and 
deployment of a “wearable computing platform” based on the sociometric badge and its machine 
analytics. The goal, the authors said, was to make machines that can “monitor social communication and 
provide real-time intervention.” To that end, twenty-two office employees were “instrumented” with the 
badge for one month in order to “automatically measure individual and collective patterns of behavior, 
predict human behavior from unconscious social signals, identify social affinity among individuals 
working in the same team, and enhance social interactions by providing feedback to the users of our 
system.” The research provided credible results, revealing patterns of communication and behavior that 
the authors concluded “would not be available without the use of a device such as the sociometric badge. 
Our results... argue strongly for the use of automatic sensing data collection tools to understand social 
systems.” They warned that organizations will become “truly sensible” only when they employ “hundreds 
or thousands of wireless environmental and wearable sensors capable of monitoring human behavior, 
extracting meaningful information, and providing managers with group performance metrics and 
employees with self-performance evaluations and recommendation.” 

The 2002 invention was continuously elaborated and eventually shepherded from lab to market. In 
2010 Pentland and his 2009 coauthors founded a company, Sociometric Solutions, to bring Skinner’s 
longed-for “instruments and methods” to the marketplace. It was one of many companies that Pentland 
would create to apply the rigors of his social physics to captive populations of office workers. 
Sociometric Solutions’ CEO, Ben Waber, one of Pentland’s doctoral students, calls his operation “people 
analytics,” and in his book of the same name, he anticipates a future of “connection, collaboration, and 
data” with the badge or something like it “deployed across millions of individuals at different companies 
in countries all over the world for not minutes but years or decades.... Imagine what we could learn about 


to help people collaborate more effectively....”2° 


Pentland and his crew continued to develop the sociometer and its applications, and by 2013 the 
device had been used by dozens of research groups and companies, including members of the Fortune 
1000. A 2014 study, authored with Waber and colleagues from Harvard and Northeastern University, 
quantified gender differences in interaction patterns. The success of the analysis occasioned this 
announcement: “It is now possible to actively instrument human behavior to collect detailed data on 
various dimensions of social interaction.” The authors signaled their aim to employ MacKay’s cardinal 
rule of unobtrusive surveillance for effective monitoring of herds, flocks, and packs, acknowledging that 
the continuous pervasive collection of human behavioral data could succeed only when conducted outside 
the boundaries of human awareness, thus eliminating possible resistance, just as we saw at Facebook. As 
the researchers enthused, “Electronic sensors can be used to complement or replace human observers 
altogether, and while they may convey a slight sense of surveillance this perception is likely reduced as 
sensors get smaller and smaller, and consequently less obtrusive.” They concluded that “minimally 
invasive ways to instrument human behavior” would enable comprehensive data collection in 
“naturalistic settings.” 

By 2015, the company opted for euphemism in a rebranding effort, changing its name to Humanyze. Its 
technology is described as a platform that uses a “smart employee badge to collect employee behavioral 
data, which it links to specific metrics with the goal of improving business performance.”“4 Waber 
portrays the work as “moneyball” for business, enabling any organization to manage its workers like a 
sports team based on measures that reveal how people move through the day, with whom they interact, 
their tone of voice, if they “lean in” to listen, their position in the social network across a variety of office 
situations, and much more, all of it to produce forty separate measures that are then integrated with a 
“business metric dashboard.” The company does not identify its client organizations, although one account 
describes its work with 10,000 employees in Bank of America’s customer service centers and a 
partnership with the consulting firm Deloitte.22 Writing in Scientific American on the power of 
sociometric data, Pentland says, “I persuaded the manager of a Bank of America call center to schedule 
coffee breaks simultaneously. The goal was to promote more engagement between employees. This single 
change resulted in a productivity increase of $15 million a year.”2° 

Of the nineteen commercial ventures currently listed in Pentland’s MIT biography, many are 
surveillance-as-a-service companies. For example, Pentland cofounded Endor, which markets itself to 
business customers as a solution to the prediction imperative. Endor’s website explains its origins in “the 
revolutionary new science” of social physics combined with a “proprietary technology” to produce a 
“powerful engine that is able to explain and predict any sort of human behavior....” The site explains that 
every human activity (e.g., phone call records, credit card purchases, taxi rides, web activity) contains a 
set of hidden mathematical patterns. On the strength of its analysis, “emerging behavioral patterns” can be 
detected before they can be observed “by any other technique.... We’ve been working with some of the 
world’s top consumer brands to unravel the most demanding of data problems.” 

In 2014 another Pentland company called Sense Networks was acquired by YP, two letters that once 
stood for “yellow pages” and now describe “North America’s largest local search, media and advertising 
company connecting consumers with local businesses.” The 2014 YP statement on its acquisition of Sense 
Networks portrays a familiar picture of the behavioral surplus land grab, describing the firm as a 
“sophisticated location data processing platform to deliver mobile audiences at scale. Sense’s retargeting 
solution for retailers can identify and reach shoppers and prospects of the top retailers with relevant 
mobile ads when they are near the retailer... at home or work.”22 

Pentland understands his experiments and paid interventions in workplace settings as emblematic of 


the larger challenges of social relations in an instrumentarian society. Once again we see the intended path 
from the economic to the social domain. Those instrumented office workers function as living labs for the 
translation of instrumentarian relations to the wider society. Pentland appeared in 2016 at a conference 
organized by Singularity University, a Silicon Valley hub of instrumentarian ideology funded in part by 
Larry Page. An interviewer tasked to write about Pentland explains, “Though people are one of the most 
valuable assets in an organization, many companies are still approaching management with a 20th century 
mentality.... Pentland saw the factor that was always messing things up was—the people.”22 Like 
Nadella, Pentland described his aims as developing the social systems that would work along the same 
lines as the machine systems, using behavioral data flows to judge the “correctness” of action patterns and 
to intervene when it is necessary to change “bad” action to “correct” action. “If people aren’t interacting 
correctly and information isn’t spreading correctly,” Pentland warns, “people make bad decisions.... 
What you’re trying to do is make a human-machine symbiote, where the humans understand more about the 
network of interactions because of the computers, and the computers are able to understand more about 
how humans work.” As the interviewer notes, “Pentland has found this data [from sociometric badges] 
goes a long way in helping organizations mend their ‘broken behaviors.’”2 

Pentland’s vision of an instrumentarian society grew in proportion to his instruments, his ideas waxing 
increasingly ambitious as the new tools and methods from his lab merged with the contemporary swell of 
computer mediation, all of it on the path to Big Other’s global ubiquity. Pentland articulated his ambitions 
for the capabilities and objectives of this new milieu in a series of papers, published primarily between 
2011 and 2014, but one remarkable 2011 essay of which he is the sole author stands out: “Society’s 
Nervous System: Building Effective Government, Energy, and Public Health Systems.” 

Pentland begins the report by announcing the institutional bona fides of this work: “Drawing on a 
unique, multi-year collaboration with the heads of major IT, wireless, hardware, health, and financial 
firms, as well as the heads of American, EU, and other regulatory organizations, and a variety of NGOs [a 
footnote here indicates the World Economic Forum], I describe the potential for pervasive and mobile 
sensing and computing over the next decade....” From there, his reasoning leaps across a range of 
inferences to stitch together a crucial rationale for a totalistic society constructed, sustained, and directed 
by instrumentarian power. The initial premise is reasonable enough: industrial-age technology once 
revolutionized the world with reliable systems for water, food, waste, energy, transportation, police, 
health care, education, and so forth, but these systems are now hopelessly “old,” “centralized,” 
“obsolete,” and “unsustainable.” 

New digital systems are required that must be “integrated,” “holistic,” “responsive,” “dynamic,” and 
“self-regulating”: “We need a radical rethinking of societies’ systems. We must create a nervous system 
for humanity that maintains the stability of our societies’ systems throughout the globe.” Referring to the 
progress of ubiquitous computational sensing devices able to govern complex machine processes and 
information flows, Pentland observes that the “sensing” technologies necessary for this nervous system 
are “already in place.” Even in 2011, Pentland understood that the basic contours of Big Other were up 
and running, describing it as a “world-spanning living organism” in which “wireless traffic systems, 
security sensors, and especially mobile telephone networks are combining to become intelligent reactive 
systems with sensors serving as their eyes and ears... the evolution... will continue at a quickening 
speed... devices will have more sensors....”=2 

But Pentland saw a problem. Although ubiquitous technologies are well on the way to solving the 
technical challenges of a global nervous system, Big Other will not be complete until it also understands 
human behavior on a global scale: “What is missing... are the dynamic models of demand and reaction,” 
along with an architecture that guarantees “safety, stability, and efficiency.... The models required must 
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describe human demand and reactions, since humans are at the core of all of these systems... the 
necessary observations are observations of individual behavior....”22 

Pentland had identified a dangerous void, which foreshadows the “profound shift” that Nadella 
extolled to Microsoft developers in 2017 when he said: “People and their relationship with other people 
is now a first-class thing in the cloud!” “People” would have to become part of Big Other’s purview, lest 
they fall prey to “incorrect” behavior. Society’s safety, stability, and efficiency hang in the balance. 
Fortunately, Pentland informs us, the instruments and methods to capture behavioral surplus for reality 
mining are uniquely suited to answer this call: 


For the first time in history, the majority of humanity is linked.... As a consequence, our mobile wireless infrastructure can be “reality 
mined” in order to... monitor our environments, and plan the development of our society.... Reality mining of the “digital 
breadcrumbs” left behind as we go about our daily lives offers potential for creating remarkable, second-by-second models of group 


dynamics and reactions over extended periods of time.... In short, we now have the capacity to collect and analyze data about people 


with a breadth and depth that was previously inconceivable.24 


In a style reminiscent of Larry Page’s rejection of “old laws,” Pentland is equally critical of a range of 
concepts and frameworks inherited from the Enlightenment and political economics. Pentland insists that 
the “old” social categories of status, class, education, race, gender, and generation are obsolete, as 
irrelevant as the energy, food, and water systems that he wants to replace. Those categories describe 
societies through the lens of history, power, and politics, but Pentland prefers “populations” to societies, 
“statistics” to meaning, and “computation” to law. He sees the “stratification of the population” coded not 
by race, income, occupation, or gender but rather by “behavior patterns” that produce “behavior 
subgroups” and a new “behavior demographics” that can predict disease, financial risk, consumer 
preferences, and political views with “between 5 and 10 times the accuracy” of the standard measures. 

A final question is urgently posed: “how to get the humans in these systems to participate in the plan?” 
His answers do not lie in persuasion or education but in behavioral modification. He says we need “new 
predictive theories of human decision making” as well as “incentive mechanism design,” an idea that is 
comparable to Skinner’s “schedules of reinforcement.” Regarding how to get humans to follow the plan, 
Pentland offers the principle of “social influence” to explain the design mechanisms through which 
millions of human beings can be herded toward the guaranteed outcomes of safety, stability, and 
efficiency. He refers to his own studies, in which “the problems of industry and government” can largely 
be explained by the pattern of information transfer, especially how people influence and mimic one 
another. 

This notion of social influence is a significant piece in Pentland’s puzzle that anticipates a great deal 
of what is to come. Pentland understands that Big Other is not only an architecture that monitors and 
controls things. Big Other’s instrumentation and data flows also make people mutually visible to one 
another, from the updates on your breakfast to the population flows in cities. Back in 2011, Pentland 
enthused, “Revolutionary new... infrastructures are providing us with a God’s eye view of ourselves.”2° 
The aim is a computer-mediated society where our mutual visibility becomes the habitat in which we 
attune to one another, producing social patterns based on imitation that can be manipulated for confluence, 
just as the logic of the machine hive suggests. 

Regarding incentives, Pentland outlines a principle of “social efficiency,” which means that 
participation must provide value to the individual but also to the system as a whole.=2 For the sake of this 
wholeness, it is believed, each of us will surrender to a totally measured life of instrumentarian order. 
Sounding ever so much like Eric Schmidt and Larry Page with their silky promises of Google’s all- 


knowing preemptive magic, Pentland believes that what we stand to lose is more than compensated for by 
the social rewards of efficient corporations and governments and the individual rewards that are simply 
magic, as he baldly appeals to second-modernity stress: 


For society, the hope is that we can use this new in-depth understanding of individual behavior to increase the efficiency and 
responsiveness of industries and governments. For individuals, the attraction is the possibility of a world where everything is arranged 
for your convenience—your health checkup is magically scheduled just as you begin to get sick, the bus comes just as you get to the 
bus stop, and there is never a line of waiting people at city hall. As these new abilities become refined by the use of more 


sophisticated statistical models and sensor capabilities, we could well see the creation of a quantitative, predictive science of human 


organizations and human society. 22 


III. The Principles of an Instrumentarian Society 


Pentland’s theory of instrumentarian society came to full flower in his 2014 book Social Physics, in 
which his tools and methods are integrated into an expansive vision of our futures in a data-driven 
instrumentarian society governed by computation. Pentland transforms Skinner’s fusty, odd utopia into 
something that sounds sophisticated, magical, and plausible, largely because it resonates with the waves 
of applied utopistics that wash over our lives each day. In completing Skinner, Pentland fashions more 
than an updated portrait of a behaviorist utopia. He outlines the principles of a full-blown instrumentarian 
society based on the pervasive outfitting and measurement of human behavior for the purposes of 
modification, control, and—in light of surveillance capitalism’s commercial dominance of the networked 
sphere—profit. 

Pentland insists that “social phenomena are really just aggregations of billions of small transactions 
between individuals....” This is a key point because it turns out that in order for social physics to replace 
the old ways of thinking, total knowledge of these billions of small things is required: “Big Data give us a 
chance to view society in all its complexity, through the millions of networks of person-to-person 
exchanges. If we had a ‘god’s eye,’ an all seeing view, then we could potentially arrive at a true 
understanding of how society works and take steps to fix our problems.” 

Pentland is sanguine on this point: total knowledge is within reach. As he states, “In just a few short 
years we are likely to have incredible rich data available about the behavior of virtually all of 
humanity—on a continuous basis. The data mostly already exists.”*° The right to the future tense—and 
with it social trust, authority, and politics—is surrendered to Big Other and the transcendent 
computational systems that rule society under the watchful eye of a group that Pentland calls “we.” He 
never defines this “we,” which imposes an us-them relationship, introducing the exclusivity of the shadow 
text and its one-way mirror. It is an omission that haunts his text. Does it refer to the priesthood of data 
scientists like Pentland? The priesthood in collaboration with the owners of the means of behavior 
modification? 

The theory aims to establish laws of social behavior comparable to laws of physics, and Pentland 
introduces two such laws that, as he says, determine the success of every “social organism.” The first is 
the quality of the “idea flow,” characterized by “exploration” to find new ideas and “engagement” to 
coordinate behavior around the best ideas. The second is “social learning,” in which people imitate one 
another until new ideas become population-wide habits. (Social learning is defined as a mathematical 
relationship derived from “how an entity’s state impacts other entities’ states and vice versa.”) Pentland 
notes that social learning is “rooted in statistical physics and machine learning.”“! The social hive is 


meant to reproduce the machine hive, and to this end Pentland advocates methods by which social 
learning “can be accelerated and shaped by social pressure.” 

The scientific aims of Pentland’s social physics depend upon a tightly integrated set of new social 
norms and individual adaptations, which I summarize here as five overarching principles that describe the 
social relations of an instrumentarian society. These principles echo Skinner’s social theory of a 
behaviorally controlled society, in which knowledge replaces freedom. In exploring each of the five, I 
compare Pentland’s statements to Skinner’s own formulations on these topics. As we shall see, Skinner’s 
once reviled thinking now defines this frontier of instrumentarian power. 


1. Behavior for the Greater Good 


Skinner had emphasized the need for an urgent shift to a collective perspective and values. “The 
intentional design of a culture and the control of human behavior it implies are essential if the human 
species is to continue to develop,” he wrote in Beyond Freedom & Dignity.’ The imperative to shift 
human behavior toward the greater good was already clear in Walden Two, where Frazier, its protagonist, 
asserts, “The fact is, we not only can control human behavior, we must.”“ Ultimately, this challenge was 
understood as an engineering problem. “And what are the techniques, the engineering practices, that will 
shape the behavior of the members of a group so that they will function smoothly for the benefit of all?” 
Frazier asks.42 Skinner advocated, via Frazier, that the virtue of a “planned society” is “to keep 
intelligence on the right track, for the good of society rather than of the intelligent individual.... It does 
this by making sure that the individual will not forget his personal stake in the welfare of society.”~® 

Pentland understands instrumentarian society as an historical turning point comparable to the invention 
of the printing press or the internet. It means that for the first time in human history, “We will have the data 
required to really know ourselves and understand how society evolves.” Pentland says that “continuous 
streams of data about human behavior” mean that everything from traffic, to energy use, to disease, to 
street crime will be accurately forecast, enabling a “world without war or financial crashes, in which 
infectious disease is quickly detected and stopped, in which energy, water, and other resources are no 
longer wasted, and in which governments are part of the solution rather than part of the problem.”“8 This 
new “collective intelligence” operates to serve the greater good as we learn to act “in a coordinated 
manner” based on “social universals.” 

“Great leaps in health care, transportation, energy, and safety are all possible,” Pentland writes, but he 
laments the obstacles to these achievements: “The main barriers are privacy concerns and the fact that we 
don’t yet have any consensus around the trade-offs between personal and social values.” Like Skinner, he 
is emphatic that these attachments to a bygone era of imperfect knowledge threaten to undermine the 
prospect of a perfectly engineered future society: “We cannot ignore the public goods that such a nervous 
system could provide....”“2 Pentland avoids the question “Whose greater g00d?” How is the greater good 
determined when surveillance capitalism owns the machines and the means of behavioral modification? 
“Goodness” arrives already oriented toward the interests of the owners of the means of behavioral 
modification and the clients whose guaranteed outcomes they seek to achieve. The greater good is 
someone’s, but it may not be ours. 


2. Plans Replace Politics 


Skinner yearned for the computational capabilities that would perfect behavioral prediction and control, 
enabling perfect knowledge to supplant politics as the means of collective decision making, In spite of his 


pre-digital limitations, Skinner had no difficulty in conceptualizing the necessary requirements for species 
salvation as a new “communal science.” As Frazier explains, “We know almost nothing about the special 
capacities of the group... the individual, no matter how extraordinary... can’t think thoughts big 
enough.” 2 

Smooth operations leave no room for unreasonable or unintentional outcomes, and Skinner viewed the 
creative and often messy conflicts of politics, especially democratic politics, as a source of friction that 
threatens the rational efficiency of the community as a single, high-functioning “superorganism.” He 
laments our inclination to try to change things with “political action,” and he endorses what he perceives 
as a widespread loss of faith in democracy. In Walden Two Frazier insists that “I don’t like the despotism 
of ignorance. I don’t like the despotism of neglect, of irresponsibility, the despotism of accident, even. 
And I don’t like the despotism of democracy!”2! 

Capitalism and socialism are equally tainted by their shared emphasis on economic growth, which 
breeds overconsumption and pollution. Skinner is intrigued by the Chinese system but rejects it on the 
grounds of the bloody revolution that any effort to convert Westerners would entail. “Fortunately,” Skinner 
concludes in the preface to Walden Two, “there is another possibility.” This option is Skinner’s version of 
a behaviorist society that provides a way in which “political action is to be avoided.” In Walden Two a 
“plan” replaces politics, overseen by a “noncompetitive” group of “Planners” who eschew power in 
favor of the dispassionate administration of the schedules of reinforcement aimed at the greater good. 
Planners exercise unique control over society but “only because that control is necessary for the proper 
functioning of the community.” 

Like Skinner, Pentland argues that computational truth must necessarily replace politics as the basis 
for instrumentarian governance. We recall Nadella’s enthusiasm over persons and relationships as 
“objects in the cloud,” when considering Pentland’s allegiance to the notion that certainty machines will 
displace earlier forms of governance. “Having a mathematical, predictive science of society that includes 
both individual differences and the relationships between individuals,” Pentland writes, “has the potential 
to dramatically change the way government officials, industry managers, and citizens think and act....”™4 

Pentland worries that our political-economic constructs such as “market” and “class” hail from an old, 
slow world of the eighteenth and nineteenth centuries. The new, “light-speed hyperconnected world” 
leaves no time for the kind of rational deliberation and face-to-face negotiation and compromise that 
characterized the social milieu in which such political concepts originated: “We can no longer think of 
ourselves as only individuals reaching carefully considered decisions; we must include the dynamic 
social effects that influence our individual decisions and drive economic bubbles, political revolutions, 
and the internet economy.”=2 

The velocity of instrumentarian society leaves us no time to get our bearings, and that speed is 
repurposed here as a moral imperative demanding that we relinquish individual agency to the automated 
systems that can keep up the pace in order to quickly perceive and impose correct answers for the greater 
good. There is no room for politics in this instrumentarian society because politics means establishing and 
asserting our bearings. Individual moral and political bearings are a source of friction that wastes 
precious time and diverts behavior from confluence. 

Instead of politics, markets, classes, and so on, Pentland reduces society to his laws of social physics: 
a reincarnation of Skinner’s “communal science.” Indeed, Pentland regards his work as the practical 
foundation of a new “computational theory of behavior” capable of producing a “causal theory of social 
structure... a mathematical explanation of why society reacts as it does and how these reactions may (or 
may not) solve human problems....” These new mathematical analyses not only reveal the deep 
“mechanisms of social interactions” (Skinner’s “special capacities of the group”) but also combine with 


“our newly acquired massive amounts of behavior data” in order to reveal the patterns of causality that 
make it possible to “engineer better social systems,” all of it based on “unprecedented instrumentation.” 

Computation thus replaces the political life of the community as the basis for governance. The depth 
and breadth of instrumentation make it possible, Pentland says, to calculate idea flow, social network 
structure, the degree of social influence between people, and even “individual susceptibilities to new 
ideas.” Most important, instrumentation makes it possible for those with the God view to modify others’ 
behavior. The data provide a “reliable prediction of how changing any of these variables will change the 
performance of all the people in the network” and thus achieve the optimum performance of Skinner’s 
superorganism. This mathematics of idea flow is the basis for Pentland’s version of a “plan” that dictates 
the targets and objectives of behavior change. Human behavior must be herded and penned within the 
parameters of the plan, just as behavior at Nadella’s construction site was continuously and automatically 
molded to policy parameters. Pentland calls this “tuning the network.” 

“Tuners” fill the role of Pentland’s “we.” He says, for example, that cities can be understood as “idea 
engines” and that “we can use the equations of social physics to begin to tune them to perform better.” 
Like Skinner’s planners, Pentland’s tuners oversee pesky anomalies that represent leakage from an old 
world of ignorance mistaken as freedom. Tuners tweak Big Other’s operations to preemptively steer such 
misguided behavior back into the fold of harmonious confluence and optimum performance for the greater 
good of whomever or whatever owns the machines that perform the math and pays the tuners to decipher 
and impose its parameters. Pentland provides an example from one of his own “living labs”: 


This mathematically derived concept of idea flow allows us to “tune” social networks in order to make better decisions and achieve 
better results.... Within the eToro digital finance world, we have found that we can shape the flows of ideas between people by 


providing small incentives or nudges to individuals, thus causing isolated traders to engage more with others and those who were too 


interconnected to engage less.... 38 


3. Social Pressure for Harmony 


In the community of Walden Two, reinforcement is precisely orchestrated to eliminate emotions that 
threaten cooperation. Only “productive and strengthening emotions—joy and love” are allowed. Feelings 
of sorrow and hate “and the high-voltage excitements of anger, fear, and rage” are considered “wasteful 
and dangerous” threats to “the needs of modern life.” Any form of distinction between persons 
undermines the harmony of the whole and its capacity to bend to collective purpose. Frazier 
acknowledges that you cannot coerce people into doing the right thing. The solution is far more subtle and 
sophisticated, based upon scientifically calibrated schedules of reinforcement: “Instead you have to set up 
certain behavioral processes which will lead the individual to design his own ‘good’ conduct.... We call 
that sort of thing ‘self-control.’ But don’t be misled, the control always rests in the last analysis in the 
hands of society.” 

Pentland’s idea is comparable: “The social physics approach to getting everyone to cooperate” is 
“social network incentives,” his version of “reinforcement.” With such incentives, he explains, “we focus 
on changing the connections between people rather than focusing on getting people individually to change 
their behavior.... We can leverage those exchanges to generate social pressure for change.”®2 Social 
media is critical to establishing these tuning capabilities, Pentland believes, because this is the 
environment in which social pressure can best be controlled, directed, manipulated, and scaled. 

In Pentland’s view Facebook already exemplifies these dynamics. Its contagion experiments reveal 
active mastery of the ability to manipulate human empathy and attachment with tuning techniques such as 


priming and suggestion. Indeed, Pentland finds Facebook’s “contagion” experiments particularly 
enlightening, seeing all sorts of practical insights in their complexities. For example, in the corporation’s 
61-million-person voting experiment, Pentland sees confirmation that social pressure can be effectively 
instrumentalized in social networks, especially among people with “strong ties”: “The knowledge that our 
face-to-face friends had already voted generated enough social pressure that it convinced people to 
vote.”®2 With this knowledge and more like it, Pentland’s “we,” the tuners, will be able to activate the 
“right incentives.” 

That Pentland’s “we” is able “to generate social pressure for change” reflects his understanding of the 
superorganism. The God view persuades him that assessing human action really is comparable to counting 
beavers: “We can observe humans in just the same way we observe apes or bees and derive rules of 
behavior, reaction, and learning.”® In all of these populations, the collective exerts pressure on each 
organism to go with the flow, stay with the herd, return to the hive, and take flight with the flock. Idea 
flows mimic the pattern of the machine hive, the edge conflates with the hub, identity yields to and 
synchrony, the parts dissolve in the whole. He writes: 


I believe that we can we think of each stream of ideas as a swarm or collective intelligence, flowing through time, with all the humans 
in it learning from each other’s experiences in order to jointly discover the patterns of preferences and habits of action that best suit 
the surrounding physical and social environment. This is counter to the way most modern Westerners understand themselves, which is 
as rational individuals, people who know what they want and who decide for themselves what actions to take in order to accomplish 


their goals.&4 


This shift from society to swarm and from individuals to organisms is the cornerstone upon which the 
structure of an instrumentarian society rests. 

Pentland ignores the role of empathy in emulation because empathy is a felt experience that is not 
subject to the observable metrics required for computational governance. Instead, Pentland subscribes to 
the label Homo imitans to convey that it is mimicry, not empathy, and certainly not politics, which defines 
human existence. The term itself derives from studies of infant learning, but for Pentland it is a fitting 
explanation of all human behavior all the time: an assertion, like Skinner’s, that control always rests with 
society. “The largest single factor driving adoption of new behaviors,” he writes, is “the behavior of 
peers.” 

Because we are born to imitate one another, Pentland argues, the whole species is attuned to social 
pressure as an efficient means of behavioral modification. This model of human learning is a throwback 
to bees and apes but also a forward pass to the machine hive. Machines do not learn by empathy; learning 
is automatically updated in a lockstep progression of collective intelligence. 


4, Applied Utopistics 


Both Skinner and Pentland believe in the authority of the utopianists to impose their plan. Instrumentarian 
society is a planned society, produced through total control of the means of behavioral modification. 
Neither Skinner’s planners nor Pentland’s tuners shrink from their responsibility to wield the power that 
shapes the superorganism. 

Skinner never lost faith in the social vision of Walden Two. He understood utopia as a “total social 
environment” in which all parts work in harmony toward collective aims: 


The home does not conflict with the school or the street, religion does not conflict with government.... And if planned economies, 


benevolent dictatorships, perfectionistic societies, and other utopian ventures have failed, we must remember that unplanned, 
undictated, and unperfected cultures have failed too. A failure is not always a mistake; it may simply be the best one can do under the 


circumstances. The real mistake is to stop trying.©© 


Pentland similarly conceives his social physics as both comprehensive and necessary. Nothing short of 
its totalistic rendering and control of all human behavior will serve civilization in a hyperconnected 
future, and there is no sign of hesitation to assert computational governance over the whole domain of 
human endeavor for the sake of a collective destiny. The politics and economics of that destiny, which is 
to say the authority and power that found and sustain it, do not require specification because the machines 
and their math transcend these once fundamental coordinates of human society. Instead, computation 
reveals the truth hidden in the data and therefore determines what is “correct.” A new social class of 
tuners exercises perpetual vigilance to cure human nature of its weaknesses by ensuring that populations 
are tuned, herded, and conditioned to produce the most-efficient behaviors. The “tools of social network 
incentives” are all that is required “to establish new norms of behavior, rather than relying on regulatory 
penalties and market competition.... Given the well-known shortcomings of human nature, social 
efficiency is a desirable goal.... Our focus should be on providing the idea flow required for individuals 
to make correct decisions and develop useful behavioral norms....”°/ Finally, like Skinner, Pentland 
rejects the notion that his imagined “data-driven society” is merely a utopian fantasy, insisting instead that 
it is not only practical and feasible but also a moral imperative in which the benefits to the collective 
outweigh all other considerations. 


5. The Death of Individuality 


Individuality is a threat to instrumentarian society, troublesome friction that sucks energy from 
“collaboration,” “harmony,” and “integration.” In an article titled “The Death of Individuality,” Pentland 
insists that “instead of individual rationality, our society appears to be governed by a collective 
intelligence that comes from the surrounding flow of ideas and examples.... It is time that we dropped the 
fiction of individuals as the unit of rationality and recognised that our rationality is largely determined by 
the surrounding social fabric....”© 

Here again the behaviorist from Harvard had already sounded the first and most eloquent iteration of 
this message, elevating the Other-One and denouncing the autonomous self. In Beyond Freedom & 
Dignity, Skinner freely displayed his contempt for this most transcendent ideal of the Sartrean age: the 
will to will oneself into first-person voice and action. Skinner argued that the differences between humans 
and other species are greatly exaggerated, and he would have found Pentland entirely justified in his 
rejection of the individual in favor of the distant, computer-mediated gaze. Beavers or people, the 
variance hardly matters once we shed the destructive fiction of individual autonomy. The surrender of the 
individual to manipulation by the planners clears the way for a safe and prosperous future built on the 
forfeit of freedom for knowledge. Skinner was unrelenting on this point: 


What is being abolished is autonomous man—the inner man, the homunculus, the possessing demon, the man defended by the 
literatures of freedom and dignity. His abolition has long been overdue.... He has been constructed from our ignorance, and as our 
understanding increases, the very stuff of which he is composed vanishes... and it must do so if it is to prevent the abolition of the 


human species. To man qua man we readily say good riddance. Only by dispossessing him can we turn... from the inferred to the 


observed, from the miraculous to the natural, from the inaccessible to the manipulable.22 


The long-overdue death of individuality finally dispels the distracting fictions that fetishize the notions 
of freedom and dignity. The twentieth-century behaviorist from Harvard and the twenty-first-century data 
scientist from MIT agree that the notion of free will is but another unfortunate hangover from a dark age 
when science had not yet demonstrated that, as Skinner says, we live “under the control of a social 
environment” that “millions of others... have constructed.” The blunt behaviorist delivers his final truth: 
“A person does not act upon the world, the world acts upon him.” 

In a lecture at Google that garnered enthusiastic applause, Pentland flattered the audience by signaling 
that the digital cognoscenti will easily accept the obsolescence of the individual as a necessary fate. 
“What about free will?” he asked the audience in Mountain View. “That may not have occurred to you, but 
that’s a traditional thing to ask.” He went on to explain that most human behavior—from political views to 
spending choices to the music that people listen to—is predicted by “what’s cool to do... exposure to 
what other people do.” Many people reject this idea, he noted, because “it’s not the rhetoric in our 
society.” Then he assured the Googlers, “You guys are the last people to be saying this to, because you 
guys are like the best and smartest in the world.” For such people as these, Pentland appeared to say, the 
death of the individual is yesterday’s news: 


So you’ve heard about rational individuals. And everybody rags on the rational part. I’m not going to do that. I’m going to rag on the 
individual part, OK? Because I don’t think we are individuals. What we desire, the ways we learn to go about doing it, what’s 


valuable, are consensual things... individual incentives... that’s part of this mindset that comes from the 1700s... the action is not 


between our ears. The action is in our social networks, OK? We are a social species.Z/ 


Pentland’s vision is Skinner’s vision, now standing on the shoulders of Big Other with its Big Data 
and its Big Math. These are the resources of the smart machines required to divine the “correct” answers. 
Such is Pentland’s resonance with Skinner’s social theory that without ever mentioning the behaviorist’s 
name, a later section of Pentland’s book is titled “Social Physics Versus Free Will and Dignity.” 

If we are to annihilate and bury the individual as an existential reality, philosophical idea, and 
political ideal, then this death should at least merit the gravitas of an ancient Greek funerary ritual. The 
existence of the individual is, after all, an achievement carved from millennia of human suffering and 
sacrifice. Instead, Pentland brushes it aside as just another debugging of humanity’s computer code, a 
much-needed upgrade to the outdated software that is the long human story. 

Unlike Skinner, though, Pentland is careful to round the square, perhaps in the hope of evading a 
withering review by the likes of Noam Chomsky. (In “The Case Against B. F. Skinner,” as you may recall 
from Chapter 10, Chomsky famously denounced Skinner as “vacuous” and “devoid of scientific content,” 
and he assessed the work as burdened with misconceptions that “virtually guarantee failure.’“) Pentland 
sidesteps the risks of Skinner’s jeremiad by taking a softer tone: “Some people react negatively to the 
phrase social physics, because they feel that it implies that people are machines without free will and 
without the ability to move independently of our role in society.” Like Meyer, Pentland acknowledges 
that humans have a “capacity for independent thought” but insists that social physics “does not need to try 
to account for it.” As Pentland sees it, the problem is not that “independent thought” is omitted from the 
picture but rather that “internal, unobservable” thought processes are just friction that “will occasionally 
emerge to defeat our best social physics models.” Fortunately, the models are not really in danger because 
“the data tell us that deviations from our regular social patterns occur only a few percent of the time.”4 
The autonomous individual is but a statistical blip, a slip of the pen that is easily overridden in the march 
toward confluent action and someone’s greater good. 

In this vein, Professor Pentland does not ignore issues like privacy and social trust. He actively 


advocates solutions to these problems, but the solutions he pursues are already tailored to the 
intensification of a “data-driven” instrumentarian society. Pentland’s approach recalls the early 
conviction of his former doctoral student Rosalind Picard that societal challenges are not insurmountable, 
that new technical solutions will resolve any problems, and that “safeguards can be developed.” Two 
decades later Picard’s view had a darker cast, but Pentland expresses little trace of doubt. For example, 
Pentland works with influential institutions like the World Economic Forum to craft “a new deal on data” 
that favors individual “ownership” of personal information but does not question the ubiquitous rendition 
of such personal information in the first instance.2 Data ownership, he believes, will create financial 
incentives for participation in a market-oriented instrumentarian society. Like Skinner, Pentland assumes 
that the sheer weight of incentives and ubiquitous connection, monitoring, and tuning will eventually wear 
down older sensibilities such as the interest in privacy. “The New Deal gives customers a stake in the 
new data economy; that will bring first greater stability and then eventually greater profitability as people 
become more comfortable sharing data.” 

In Pentland’s view of data ownership, certainty machines like blockchain, which relies on complex 
encryption and algorithms to create a decentralized tamper-proof database, are commandeered to bypass 
social trust. He advocates systems “that live everywhere and nowhere, protecting and processing the data 
of millions of people, and executing on millions of internet computers.” One important study of Bitcoin, 
the cryptocurrency that relies on blockchain, suggests that such machine solutions both express and 
contribute to the general erosion of the social fabric in ways that are both consistent with 
instrumentarianism and further pave the way for its success. Information scholars Primavera De Filippi 
and Benjamin Loveluck conclude that contrary to popular belief, “Bitcoin is neither anonymous nor 
privacy-friendly... anyone with a copy of the blockchain can see the history of all Bitcoin transactions... 
every transaction ever done on the Bitcoin network can be traced back to its origin.” Such systems rely on 
“perfect information,” but the kinds of coordination processes that build open democratic societies, such 
as “social trust” or “loyalty,” are “expunged” in favor of “a profoundly market-driven approach.” Like 
Varian, Pentland does not acknowledge the social and political implications of such systems, which are in 
any case irrelevant to an instrumentarian future in which democracy and social trust are superseded by the 
certainty machines, their priests, and their owners. 

Surveillance capitalism grew to dominance during the years that Pentland has argued for his “New 
Deal,” even as it benefitted from his theoretical and commercial innovations. During those same years, as 
we have seen, Picard’s “affective computing” fell to the surveillance paradigm. Nevertheless, Professor 
Pentland is sanguine that surveillance capitalism can easily be pushed aside by market forces, despite its 
concentrations of knowledge, rights, and power; its unilateral control of the shadow text; and its dominant 
position in the division of learning in society. “It simply requires that creative businesspeople harness the 
will of consumers in order to construct a value proposition better than the current steal-all-your-data 
paradigm. We’ve just got to push on through.” Power, politics, and law do not enter into the equation, 
presumably because they are already obsolete in the social vision under construction here. 


IV. The Third Modernity of the Hive 


It is no surprise that capitalism shapes social relations. A century ago it was the new means of mass 
production that fashioned mass society in its image. Today, surveillance capitalism offers a new template 
for our future: the machine hive in which our freedom is forfeit to perfect knowledge administered for 
others’ profit. This is an unheralded social revolution that is difficult to discern in the fog of utopian 


rhetoric and high-speed applied utopistics conjured by leading surveillance capitalists and the many 
communities of practice—from developers to data scientists—that enable and sustain the dominance of 
the commercial surveillance project. 

Surveillance capitalists work hard to camouflage their purpose as they master the uses of 
instrumentarian power to shape our behavior while evading our awareness. That is why Google conceals 
the operations that turn us into the objects of its search and Facebook distracts us from the fact that our 
beloved connections are essential to the profit and power that flow from its network ubiquity and 
totalistic knowledge. 

Pentland’s experimental work and theoretical analyses perform an important political and social 
function in piercing this fog. They map the tactical and conceptual pathways of instrumentarian society 
that place the means of behavior modification at the heart of this social system, founded on the scientific 
and technological control of collective behavior and administered by a specialist class. In China the state 
appears determined to “own” this complex, but in the West it is largely owned and operated by 
surveillance capital. 

Instrumentarian society defines the ultimate institutionalization of a pathological division of learning. 
Who knows? Who decides? Who decides who decides? Here too the comparison with China is useful. An 
abnormal division of learning marks both China and the West. In China the state vies with its surveillance 
capitalists for control. In the US and Europe the state works with and through the surveillance capitalists 
to accomplish its aims. It is the private companies who have scaled the rock face to command the heights. 
They sit at the pinnacle of the division of learning, having amassed unprecedented and exclusive wealth, 
information, and expertise on the strength of their dispossession of our behavior. They are making their 
dreams come true. Not even Skinner could have aspired to this condition. 

The social principles of instrumentarianism’s third modernity represent a stark break with the legacies 
and ideals of the liberal order. Instrumentarian society is a topsy-turvy fun-house-mirror world in which 
everything that we have cherished is turned upside down and inside out. Pentland doubles down on the 
illiberality of behavioral economics. In his hands the ideology of human frailty is not merely cause for 
contempt but a justification for the death of individuality. Self-determination and autonomous moral 
judgment, generally regarded as the bulwark of civilization, are recast as a threat to collective well- 
being. Social pressure, well-known to psychologists for its dangerous production of obedience and 
conformity, is elevated to the highest good as the means to extinguish the unpredictable influences of 
autonomous thought and moral judgment. 

These new architectures feed on our fellow feeling to exploit and ultimately to suffocate the 
individually sensed inwardness that is the wellspring of personal autonomy and moral judgment, the first- 
person voice, the will to will, and the sense of an inalienable right to the future tense. That we vibrate to 
one another should be a life-enhancing fact, but this third modernity amplifies our mutual vibration to an 
excruciating pitch. In the milieu of total instrumentation, it is less that we resonate to one another’s 
presence and more that we drown in its inescapability. 

Instrumentarianism reimagines society as a hive to be monitored and tuned for guaranteed outcomes, 
but this tells us nothing of the lived experience of its members. What are the consequences of life lived in 
the hive, where one is perceived as an “other” to the surveillance capitalists, designers, and tuners who 
impose their instruments and methods? How and when do we each become an organism among organisms 
to ourselves and to one another, and with what result? The answers to these questions are not all 
guesswork. We can begin by asking our children. Without knowing it, we sent the least formed and most 
vulnerable among us to scout the hive and settle its wilderness. Now their messages are filtering in from 
the frontier. 


CHAPTER SIXTEEN 


OF LIFE IN THE HIVE 


All grew so fast his life was overgrown, 
Till he forgot what all had once been made for: 
He gathered into crowds but was alone... 


—W. H. AUDEN 
SONNETS FROM CHINA, VIII 


I. Our Canaries in the Coal Mine 


“T felt so lonely... I could not sleep well without sharing or connecting to others,” a Chinese girl recalled. 
“Emptiness,” an Argentine boy moaned. “Emptiness overwhelms me.” A Ugandan teenager muttered, “I 
felt like there was a problem with me,” and an American college student whimpered, “I went into 
absolute panic mode.” These are but a few of the lamentations plucked from one thousand student 
participants in an international study of media use that spanned ten countries and five continents. They had 
been asked to abstain from all digital media for a mere twenty-four hours, and the experience released a 
planet-wide gnashing of teeth and tearing of flesh that even the study’s directors found disquieting.’ 
Capping the collective cri de coeur, a Slovakian university student reflected, “Maybe it is unhealthy that I 
can’t be without knowing what people are saying and feeling, where they are, and what’s happening.” 

The students’ accounts are a message in a bottle for the rest of us, narrating the mental and emotional 
milieu of life in an instrumentarian society with its architectures of behavioral control, social pressure, 
and asymmetrical power. Most significantly, our children are harbingers of the emotional toll of the 
viewpoint of the Other-One as young people find themselves immersed in a hive life, where the other is 
an “it” to me, and I experience myself as the “it” that others see. These messages offer a glimpse of the 
instrumentarian future, like the scenes revealed by Dickens’s Ghost of Christmas Yet to Come. So shaken 
was Scrooge by his glimpse of bitter destiny that he devoted the remainder of his life to altering its 
course. What will we do? 

The question hangs over this chapter. Pentland celebrates Facebook as the perfect milieu for effective 
social pressure and tuning. In the sections that follow, we explore the mechanisms that Pentland admires. 
Why is it so difficult for young people to unplug? What are the consequences of that attachment for them 
and for all of us? Facebook has learned to bite hard on the psychological needs of young people, creating 
new challenges for the developmental processes that build individual identity and personal autonomy. The 


effects of these challenges are already evident in a parade of studies that document the emotional toll of 
social media on young people. As we shall see, the hive and its larger architecture of Big Other plunge us 
into an intolerable world of “no exit.” 

The international “unplug” study helps to set the stage, for it reveals a range of emotional anguish 
summarized in six categories: addiction, failure to unplug, boredom, confusion, distress, and isolation. 
The students’ sudden disconnection from the network produced the kinds of cravings, depression, and 
anxiety that are characteristic of clinically diagnosed addictions. The result was that a majority in every 
country admitted that they could not last out the day unplugged. Their angst was compounded by the same 
Faustian pact with which we are all too familiar, as they discovered that nearly all daily logistical, 
communicative, and informational requirements were dependent upon their connected devices: “Meeting 
with friends became difficult or impossible, finding the way to a destination without an online map or 
access to the internet became a problem, and simply organizing an evening at home became a challenge.” 
Worse yet, the students found it impossible to imagine even casual social participation without social 
media, especially Facebook: “Increasingly no young person who wants a social life can afford not to be 
active on the site, and being active on the site means living one’s life on the site.” 

Business and tech analysts cite “network effects” as a structural source of Facebook’s dominance in 
social media, but those effects initially derived from the demand characteristics of adolescents and 
emerging adults, reflecting the peer orientation of their age and stage. Indeed, Facebook’s early advantage 
in this work arose in no small measure from the simple fact that its founders and original designers were 
themselves adolescents and emerging adults. They designed practices for an imagined universe of 
adolescent users and college students, and those practices were later institutionalized for the rest of us, 
reducing the social world to a tally of “friends” who are not friends and “likes” that provide a continuous 
ticker tape of one’s value on the social market, stoking the anxieties of pre-adulthood and anticipating the 
mesmerizing social disciplines of the hive.? 

The researchers concluded that their global study of students had “ripped back the curtain” on the 
loneliness and acute disorientation that overwhelm young people when faced with disconnection from 
social media. It wasn’t simply that they didn’t know what to do with themselves but rather that “they had 
problems articulating what they were feeling or even who they were if they couldn’t connect.” The 
students felt as though “they had lost part of themselves.”2 

These feelings of disorientation and isolation suggest a psychological dependency on the “others,” and 
additional studies only deepen our understanding of how “Generation Z,” the demographic cohort born in 
and after 1996—the first group of digital natives, with no memory of life before the rise of surveillance 
capitalism—relies on a range of social media for psychological sustenance as they bounce between four 
or five platforms more or less simultaneously. Consider first the older cohorts. A 2012 survey concluded 
that emerging adults devote more time to using media than any other daily activity, spending nearly twelve 
hours each day with media of some form.4 By 2018 Pew Research reported that nearly 40 percent of 
young people ages 18—29 report being online “almost constantly,” as do 36 percent of those ages 30-49. 
Generation Z intensifies the trend: 95 percent use smartphones, and 45 percent of teens say they are online 
“on a near-constant basis.”2 If that is how you spend your days and nights, then the findings of a 2016 
study are all too logical, as 42 percent of teenage respondents said that social media affects how people 
see them, having adopted what the researchers call an outside-looking-in approach to how they express 
themselves. Their dependency penetrates deeply into their sense of well-being, affecting how they feel 
about themselves (42 percent) and their happiness (37 percent).° 

In a subsequent elaboration on the psychological consequences of experiencing oneself from the 
“outside looking in,” a 2017 survey of young British women ages 11—21 suggests that the social principles 


of instrumentarian society, so enthusiastically elaborated by Pentland and endorsed by surveillance 
capitalist leaders, appear to be working effectively.2 Thirty-five percent of the women said that their 
biggest worry online was comparing themselves and their lives with others as they are drawn into 
“constant comparisons with often idealized versions of the lives, and bodies, of others.’”2 

A director of the project observed that even the youngest girls in this cohort feel pressured to create a 
“personal brand,” the ultimate in self-objectification, as they seek reassurance “in the form of likes and 
shares.” When the Guardian tried to explore girls’ reflections on these survey findings, the responses 
eloquently betray the plight of the organism among organisms. “I do feel I need to be perfect and compare 
myself to others all the time,” says one. “You see other people’s lives and what they are doing... you... 
see their ‘perfect’ lives and it makes you think yours isn’t,” says another.? 

In light of these findings, one UK medical specialist comments on the young people in her practice: 
“People are growing up to want to be influencers and that is now a job role.... Iam not sure if parents are 
fully aware of the pressure people face....”42 Indeed, only 12 percent of respondents in that 2017 survey 
reckoned that their parents understood these pressures. The reports confirm that social pressure is well 
institutionalized as the means of online social influence, but contrary to Pentland’s belief that “class” 
divisions would disappear, life in the hive produces new cleavages and forms of stratification: not only 
tune or be tuned but also pressure or be pressured. 

Nothing summarizes young life in the hive better than the insights of Facebook’s own North American 
marketing director, Michelle Klein, who told an audience in 2016 that while the average adult checks his 
or her phone 30 times a day, the average millennial, she enthusiastically reported, checks more than 157 
times daily. Generation Z, we now know, exceeds this pace. Klein described Facebook’s engineering feat: 
“a sensory experience of communication that helps us connect to others, without having to look away,” 
noting with satisfaction that this condition is a boon to marketers. She underscored the design 
characteristics that produce this mesmerizing effect: design is narrative, engrossing, immediate, 
expressive, immersive, adaptive, and dynamic. 

If you are over the age of thirty, you know that Klein is not describing your adolescence, or that of your 
parents, and certainly not that of your grandparents. Adolescence and emerging adulthood in the hive are a 
human first, meticulously crafted by the science of behavioral engineering; institutionalized in the vast and 
complex architectures of computer-mediated means of behavior modification; overseen by Big Other; 
directed toward economies of scale, scope, and action in the capture of behavioral surplus; and funded by 
the surveillance capital that accrues from unprecedented concentrations of knowledge and power. Our 
children endeavor to come of age in a hive that is owned and operated by the applied utopianists of 
surveillance capitalism and is continuously monitored and shaped by the gathering force of 
instrumentarian power. Is this the life that we want for the most open, pliable, eager, self-conscious, and 
promising members of our society? 


II. The Hand and the Glove 


The magnetic pull that social media exerts on young people drives them toward more automatic and less 
voluntary behavior. For too many, that behavior shades into the territory of genuine compulsion. What is it 
that mesmerizes the youngest among us, lashing them to this mediated world despite the stress and disquiet 
that they encounter there? 

The answer lies in a combination of behavioral science and high-stakes design that is precision-tooled 
to bite hard on the felt needs of this age and stage: a perfectly fitted hand and glove. Social media is 


designed to engage and hold people of all ages, but it is principally molded to the psychological structure 
of adolescence and emerging adulthood, when one is naturally oriented toward the “others,” especially 
toward the rewards of group recognition, acceptance, belonging, and inclusion. For many, this close 
tailoring, combined with the practical dependencies of social participation, turns social media into a toxic 
milieu. Not only does this milieu extract a heavy psychological toll, but it also threatens the course of 
human development for today’s young and the generations that follow, all spirits of a Christmas Yet to 
Come. 

The hand-and-glove relationship of technology addiction was not invented at Facebook, but rather it 
was pioneered, tested, and perfected with outstanding success in the gaming industry, another setting 
where addiction is formally recognized as a boundless source of profit. Skinner had anticipated the 
relevance of his methods to the casino environment, which executives and engineers have transformed 
into as vivid an illustration as one can muster of the startling power of behavioral engineering and its 
ability to exploit individual inclinations and transform them into closed loops of obsession and 
compulsion. 

No one has mapped the casino terrain more insightfully than MIT social anthropologist Natasha Dow 
Schiill in her fascinating examination of machine gambling in Las Vegas, Addiction by Design. Most 
interesting for us is her account of the symbiotic design principles of a new generation of slot machines 
calculated to manipulate the psychological orientation of players so that first they never have to look 
away, and eventually they become incapable of doing so. Schiill learned that addictive players seek 
neither entertainment nor the mythical jackpot of cash. Instead, they chase what Harvard Medical School 
addiction researcher Howard Shaffer calls “the capacity of the drug or gamble to shift subjective 
experience,” pursuing an experiential state that Schiill calls the “machine zone,” a state of self-forgetting 
in which one is carried along by an irresistible momentum that feels like one is “played by the 
machine.” The machine zone achieves a sense of complete immersion that recalls Klein’s description of 
Facebook’s design principles—engrossing, immersive, immediate—and is associated with a loss of self- 
awareness, automatic behavior, and a total rhythmic absorption carried along on a wave of compulsion. 
Eventually, every aspect of casino machine design was geared to echo, enhance, and intensify the hunger 
for that subjective shift, but always in ways that elude the player’s awareness. 

Schiill describes the multi-decade learning curve as gaming executives gradually came to appreciate 
that a new generation of computer-based slot machines could trigger and amplify the compulsion to chase 
the zone, as well as extend the time that each player spends in the zone. These innovations drive up 
revenues with the sheer volume of extended play as each machine is transformed into a “personalized 
reward device.”2 The idea, as the casinos came to understand it, is to avoid anything that distracts, 
diverts, or interrupts the player’s fusion with the machine; consoles “mold to the player’s natural 
posture,” eliminating the distance between the player’s body and frictionless touch screens: “Every 
feature of a slot machine—its mathematical structure, visual graphics, sound dynamics, seating and 
screen ergonomics—is calibrated to increase a gambler’s ‘time on device’ and to encourage ‘play to 
extinction.’”4 The aim is a kind of crazed machine sex, an intimate closed-loop architecture of 
obsession, loss of self, and auto-gratification. The key, one casino executive says in words that are all too 
familiar, “is figuring out how to leverage technology to act on customers’ preferences [while making] it as 
invisible—or what I call auto-magic—as possible.” 

The psychological hazards of the hand-glove fit have spread far beyond the casino pits where players 
seek the machine zone: they define the raw heart of Facebook’s success. The corporation brings more 
capital, information, and science to this parasitic symbiosis than the gaming industry could ever muster. Its 
achievements, pursued in the name of surveillance revenues, have produced a prototype of 


instrumentarian society and its social principles, especially for the youngest among us. There is much that 
we can grasp about the lived experience of the hive in the challenges faced by the young people whose 
fate it is to come of age in this novel social milieu in which the forces of capital are dedicated to the 
production of compulsion. Facebook’s marketing director openly boasts that its precision tools craft a 
medium in which users “never have to look away,” but the corporation has been far more circumspect 
about the design practices that eventually make users, especially young users, incapable of looking away. 

There are some chinks in the armor. For example, in 2017 Napster cofounder and one-time Facebook 
president Sean Parker frankly admitted that Facebook was designed to consume the maximum possible 
amount of users’ time and consciousness. The idea was to send you “a little dopamine hit every once ina 
while”—a.k.a. “variable reinforcement—in the form of ‘likes’ and comments. The goal was to keep users 
glued to the hive, chasing those hits while leaving a stream of raw materials in their wake.” 1° 

Shaffer, the addiction researcher, has identified five elements that characterize this state of 
compulsion: frequency of use, duration of action, potency, route of administration, and player attributes. 
We already know quite a bit about the high frequency and long duration of young people’s engagement in 
social media. What we need to understand is something of (1) the psychological attributes that draw them 
to social media in the first place (the hand), (2) the design practices that ratchet up potency in order to 
transform inclinations into unquenchable need (the glove), and (3) the mental and emotional consequences 
of Facebook’s ever-more-exquisite ability to enmesh young people in chasing their own kind of zone. 

Consider the final moments of a 2017 Washington Post profile on a thirteen-year-old girl, part of a 
series chronicling “what it’s like to grow up in the age of likes, lols, and longing.” It is the girl’s birthday, 
and only one question will decide her happiness: do her friends like her enough to post pictures of her on 
their pages in appreciation of the occasion? “She scrolls, she waits. For that little notification box to 
appear.”/2 Regardless of your age, who among us does not feel a painful blast of recognition? 
Adolescence has always been a time when acceptance, inclusion, and recognition from the “others” can 
feel like matters of life and death, and social media has not been required to make it so. Is adolescence 
really any different today than in any other era? The answer is yes... and no. 

Adolescence was officially “discovered” in the United States in 1904 by G. Stanley Hall, and even 
then, Hall, the first doctor of psychology in the country, located the challenges of youth in the rapidly 
changing context of “our urbanized hothouse life that tends to ripen everything before its time.”!8 While 
writing about teenagers in 1904, he observed that adolescence is a period of extreme orientation toward 
the peer group: “Some seem for a time to have no resource in themselves, but to be abjectly dependent for 
their happiness upon their mates.”!2 He also pointed to the potential for cruelty within the peer group, a 
phenomenon that contemporary psychologists refer to as “relational aggression.” Decades later, the 
central challenge of adolescence was famously characterized as “identity formation” by the 
developmental psychologist Erik Erikson, who did much to explain twentieth-century adolescence. 
Erikson emphasized the adolescent struggle to construct a coherent identity from the mutual “joinedness” 
of the adolescent clique. He described the “normative crisis” when fundamental questions of “right” and 
“wrong” require inner resources associated with “introspection” and “personal experimentation.” The 
healthy resolution of that conflict between self and other leads to a durable sense of identity.72 

Today, most psychologists agree that our longer lives combined with the challenges of an information- 
intensive society have further lengthened the time between childhood and adulthood. Many have settled on 
the notion of “emerging adulthood” to denote the years between eighteen and the late twenties as a new 
life stage: emerging adulthood is to the twenty-first century what adolescence was to the twentieth.2! And 
although contemporary researchers embrace a diverse range of methods and paradigms, most concur that 
the essential challenge of emerging adulthood is the differentiation of a “self” from the “others.”22 


There is a broad consensus that our extended life spans often require us to revisit the core questions of 
identity more than once during our lives, but researchers agree that psychological success during emerging 
adulthood depends on at least some resolution of identity issues as the basis for the shift toward full 
adulthood. As one research scholar writes, “A prime challenge of emerging adulthood is to become the 
author of your own life.”22 Who among us does not recognize that call? This existential challenge is 
enduring, a source of continuity that links generations. What has changed are the circumstances in which 
young people today must meet this challenge. 


III. Proof of Life 


Emerging adulthood is “ground zero” in the struggle for the “relational autonomy” that prepares young 
people for the transition into adulthood, as Notre Dame psychologists Daniel Lapsley and Ryan Woodbury 
characterize it.24 By “relational autonomy,” they mean to underscore the idea that autonomy is not a 
simplistic cliché of “individualism,” unencumbered by attachment or empathy, but instead it strikes a vital 
balance between the cultivation of inner resources and the capacity for intimacy and relationship. 
Emerging adulthood requires “hard bargaining” to establish a self that is separate from but still connected 
to others, and the quality of this inner bargain “gives emerging adulthood a sense of anticipation and 
urgency,” aiding a successful transition to adulthood. 

Even with these insights, it remains difficult to fully grasp the felt experiences of young people who, 
as Hall aptly described more than a century ago, “seem... to have no resources in themselves.” Perhaps 
the most difficult quality to capture is that in this period that precedes the hard bargaining, an “inner” 
sense of “self” simply does not yet exist. It is a time when “I” am whatever the “others” think of me, and 
how “T” feel is a function of how the “others” treat me. Instead of a stable sense of identity, there is only a 
chameleon that reinvents itself depending upon the social mirror into which it is drawn. In this condition, 
the “others” are not individuals but the audience for whom I perform. Who “I” am depends upon the 
audience. This state of existence in the mirror is pure “fusion,” and it captures the meaning of a thirteen- 
year-old girl anxiously awaiting the appearance of the little notification box as a sign of her existence and 
her worth. The young person who has not yet carved out an inward space exists for herself only in the 
viewpoint of the Other-One. Without the “others,” the lights go out. Anger is out of the question: one dare 
not alienate the others who are one’s mirror and thus one’s proof of life. 

In this most elemental sense, the young person who feels compelled to use social media is more truly 
and accurately described as hanging on for dear life, alive in the gaze of others because it’s the only life 
one has, even when it hurts. As developmental psychologist Robert Kegan described the adolescent 
experience long before the advent of Facebook, “There is no self independent of the context of ‘other 
people liking.’”2° This is not a moral or emotional shortcoming but a fact of life in this developmental 
moment, and it entails certain predictable consequences. For example, one tends to operate through social 
comparison. One can be easy prey to manipulation, with few defenses against social pressure and other 
forms of social influence. The fixed belief system of an established group can all too easily fill the inner 
void, substituting an externally sourced identity for the work of self-construction.22 

Moving on from “fusion” means a transition from being someone who is their relationships to someone 
who has their relationships. It entails a deep reconfiguration of how we make sense of our experience. In 
Kegan’s language, this means a shift away from a “culture of mutuality” to a more complex “culture of 
identity, self-authorship, and personal autonomy.” This shift depends upon encountering people and life 
experiences that demand something more than our reflection in the mirror. It requires individuals and 


situations that insist on our first-person voice, provoking us to carve out our own unique response to the 
world. 

This is an inner act that eludes rendition or datafication as we begin to compose an inward sense of 
valid truth and moral authority. This is the reference point from which we can say, “I think,” “I feel,” “I 
believe.” Gradually, this “I” learns to feel authorship and ownership of its experiences. It can reflect on 
itself, know itself, and regulate itself with intentional choices and purposive action. Research shows that 
these big leaps in self-construction are stimulated by experiences such as structured reflection, conflict, 
dissonance, crisis, and failure. The people who help trigger this new inward connection refuse to act as 
our mirrors. They reject fusion in favor of genuine reciprocity. “Who comes into a person’s life,” Kegan 
observes, “may be the single greatest factor to influence what that life becomes.”2 

What are the consequences of the failure to win a healthy balance between inner and outer, self and 
relationship? Clinical studies identify specific patterns associated with this developmental stagnation. 
Not surprisingly, these include an inability to tolerate solitude, the feeling of being merged with others, an 
unstable sense of self, and even an excessive need to control others as a way of keeping the mirror close. 
Loss of the mirror is the felt equivalent of extinction.“2 

The cultivation of inner resources is thus critical to the capacity for intimacy and relationship, 
challenges that have become more time-consuming with each new phase of the modern era. And while 
young people are bound as ever to the enduring existential task of self-making, our story suggests three 
critical ways in which this task now converges with history and the unique conditions of existence in our 
time. 

First, the waning of traditional society and the evolution of social complexity have accelerated the 
processes of individualization. We must rely upon our self-making and inner resources more than at any 
time in the human story, and when these are thwarted, the sense of dislocation and isolation is bitter. 

Second, digital connection has become a necessary means of social participation, in part because of a 
widespread institutional failure to adapt to the needs of a new society of individuals. The computer 
mediation of the social infrastructure simultaneously alters human communication, illuminating individual 
and collective behavior, as reflected in the undulating waves of tweets, likes, clicks, patterns of mobility, 
search queries, posts, and thousands of other daily actions. 

Third, surveillance capitalism dominates and instrumentalizes digital connection. “What is different as 
a result of social media,” writes researcher danah boyd in her examination of the social lives of 
networked teens, “is that teens’ perennial desire for social connection and autonomy is now being 
expressed in networked publics.”=2 It’s true that for the sake of connection, the travails of identity are 
visible to a wider group. But the notion of “networked publics” is a paradox. In fact, our visibility is 
magnified and compelled not only by the public-ness of networked spaces but by the fact that they are 
privatized. Young life now unfolds in the spaces of private capital, owned and operated by surveillance 
capitalists, mediated by their “economic orientation,” and operationalized in practices designed to 
maximize surveillance revenues. These private spaces are the media through which every form of social 
influence—social pressure, social comparison, modeling, subliminal priming—is summoned to tune, 
herd, and manipulate behavior in the name of surveillance revenues. This is where adulthood is now 
expected to emerge. 

Whereas casino executives and slot machine developers can be garrulous and boastful, eager to share 
their “addiction by design” achievements, the surveillance capitalist project relies on secrecy. An entire 
discourse has thus sprung to life, trained on decoding the stealth design that first deters users from ever 
looking away and then makes them incapable of doing so. There are chat groups and endless query threads 
as people try to divine what Facebook actually does. Relevant design practices are discussed in 


journalistic accounts as well as in books with such titles as Evil by Design, Hooked, and Irresistible, all 
of which help to normalize the very methods they discuss. For example, Evil by Design author Chris 
Nodder, a user-experience consultant, explains that evil design aims to exploit human weakness by 
creating interfaces that “make users emotionally involved in doing something that benefits the designer 
more than them.” He coaches his readers in psychic numbing, urging them to accept the fact that such 
practices have become the standard suggesting that consumers and designers find ways to “turn them to 
your advantage.” 

If we are to judge coming of age in our time, then we have to understand something of the specific 
practices that turn social participation into a glove that doesn’t simply embrace the hand but rather 
magnetizes and paralyzes the hand for the sake of economic imperatives. Facebook relies on specific 
practices that feed the inclinations of people, especially young people, to know themselves from “the 
outside looking in.” Most critical is that the more the need for the “others” is fed, the less able one is to 
engage the work of self-construction. So devastating is the failure to attain that positive equilibrium 
between inner and outer life that Lapsley and Woodbury say it is “at the heart” of most adult personality 
disorders.°2 

For example, Nodder highlights Facebook’s precocious mastery of “social proof’: “Much of our 
behavior is determined by our impressions of what is the correct thing to do... based on what we observe 
others doing.... This influence is known as social proof.”2? The company instrumentalizes this aspect of 
adolescent nature by using messages from “friends” to make a product, service, or activity feel “more 
personal and emotional.” This ubiquitous tactic, much admired by Pentland, was used in the Facebook 
voting experiment. It fuels young people’s needs to garner approval and avoid disapproval by doing what 
the others are doing. 

Facebook’s single most momentous innovation in behavioral engineering is the now equally ubiquitous 
“Like” button, adopted in 2009. According to contemporaneous blog posts by longtime Facebook 
executive Andrew Bosworth, the “Like” button had been debated internally for more than a year and a 
half before Zuckerberg’s final decision to incorporate it. He had rejected the idea more than once, fearing 
that it would detract from other features intended to lift monetization, such as the controversial Beacon 
program. Significantly, the founder embraced the button only when new data revealed it as a powerful 
source of behavioral surplus that helped to ratchet up the magnetism of the Facebook News Feed, as 
measured by the volume of comments. 

Facebook’s leadership appears to have realized only gradually that the button could transform the 
platform from a book into a blizzard of mirrors, a passive read into an active sea of mutual reflections that 
would glue users to their news feeds. On the supply side, the “Like” button was a planet-size one-way 
mirror capable of exponentially increasing raw-material supplies. The more that a user “liked,” the more 
that she informed Facebook about the precise shape and composition of her “hand,” thus allowing the 
company to continuously tighten the glove and increase the predictive value of her signals. 

The protocols at Instagram, a Facebook property, provide another good example of these processes. 
Here one sees these tight linkages as compulsion draws more surplus to feed more compulsion. Instagram 
rivets its users with photos that appeal to their interests, so how does it select those photos from the 
millions that are available? The obvious, but incorrect, answer would be that it analyzes the contents of 
photos that you like and shows you more. Instead, Instagram’s analytics are drawn from behavioral 
surplus: the shadow text. As one manager describes it, “You base predictions off an action, and then you 
do stuff around that action.” Actions are signals like “following,” “liking,” and “sharing,” now and in the 
past. The circle widens from there. With whom did you share? Who do they follow, like, and share with? 
“Instagram is mining the multilayered social web between users,” but that mining is based on observable, 


measurable behaviors moving through time: the dynamic surplus of the shadow text drawn from its own 
caches as well as Facebook’s, not the content displayed in the public text.*° In the end, the photos you see 
resonate with strange relevance for your life. More begets more. 

On the demand side, Facebook’s “likes” were quickly coveted and craved, morphing into a universal 
reward system or what one young app designer called “our generation’s crack cocaine.” “Likes” became 
those variably timed dopamine shots, driving users to double down on their bets “every time they shared a 
photo, web link, or status update. A post with zero likes wasn’t just privately painful, but also a kind of 
public condemnation.”=° In fact, most users craved the reward more than they feared humiliation, and the 
“Like” button became Facebook’s signature, spreading across the digital universe and actively fusing 
users in a new kind of mutual dependency expressed in a pastel orgy of giving and receiving 
reinforcement. 

The “Like” button was only the start of what was to be an historic construction of a new social world 
that for many users is defined by fusion with the social mirror, especially among the young. Just as 
gamblers chase the zone of fusion with the machine, a young person embedded in the culture of mutuality 
chases the zone of fusion with the social mirror. For anyone already struggling with the challenge of the 
self-other balance, the “Like” button and its brethren continuously tip the scales toward regression. 

The short history of Facebook’s News Feed is further evidence of the efficacy of the ever-tightening 
feedback loops that aim to shape and sustain this fusion. When News Feed was first launched in 2006, it 
transformed Facebook from a site where users had to visit friends’ pages to see their updates to having 
those messages automatically shared in a stream on each person’s home page. Hundreds of thousands of 
users joined opposition groups, repelled by the company’s unilateral invasion of privacy. “No one was 
prepared for their online activity to suddenly be fodder for mass consumption,” recalled the tech news 
site TechCrunch on News Feed’s tenth anniversary in 2016, as it offered readers “The Ultimate Guide to 
the News Feed,” with instructions on “how you can get your content seen by more people,” how to appear 
“prominently,” and how to resonate with your “audience.”=2 Ten years earlier a TechCrunch reporter had 
presciently noted, “Users who don’t participate will quickly find that they are falling out of the attention 
stream, and I suspect will quickly add themselves back in.” 

Playing to the fear of invisibility and abandonment worked in 2006, when Facebook had just 9.5 
million users (and required a college e-mail address to sign up), and it has driven the acceptance of every 
subsequent tweak to News Feed as Facebook has amassed more than 2 billion users. News Feed grew to 
become the “epicenter” of the corporation’s revenue success and “the most valuable billboard on Earth,” 
as Time magazine stated in 2015, just three years after Facebook’s IPO.” 

News Feed is also the fulcrum of the social mirror. In the years between revulsion and reverence, 
News Feed became Facebook’s most intensely scrutinized object of data science and the subject of 
extensive organizational innovation, all of it undertaken at a level of sophistication and capital intensity 
that one might more naturally associate with the drive to solve world hunger, cure cancer, or avert climate 
destruction. 

In addition to Facebook’s already complex computational machinery for targeting ads, by 2016 the 
News Feed function depended upon one of the world’s most secretive predictive algorithms, derived 
from a God view of more than 100,000 elements of behavioral surplus that are continuously computed to 
determine the “personal relevancy” score of thousands of possible posts as it “scans and collects 
everything posted in the past week by each of your friends, everyone you follow, each group you belong 
to, and every Facebook page you’ve liked,” writes Will Oremus in Slate. “The post you see at the top of 
your feed, then, has been chosen over thousands of others as the one most likely to make you laugh, cry, 
smile, click, like, share, or comment.”“2 The glove tightens around the hand with closed feedback loops 


enabled by the God view, which favors posts from people with whom you have already interacted, posts 
that have drawn high levels of engagement from others, and posts that are like the ones with which you 
have already engaged.+! 

In 2015 the See First “curation tool” was introduced to channel direct data on the shape of a user’s 
social mirror by soliciting his or her personal priorities for the News Feed. Facebook’s chief product 
officer describes the corporation’s interest in supplying what is “most meaningful” for you to know today 
from “everything that happened on Earth... published anywhere by any of your friends, any of your family, 
any news source.”4* Each post sequenced in the News Feed also now hosts a range of explicit feedback 
options: I want more of this. I want none of that. These direct surplus supply lines are important sources 
of innovation aimed at broadening the target of the fusion zone, increasing the tenacity of an ever- 
tightening glove. In 2016 Facebook’s product director confirmed that this direct sourcing of surplus “led 
to an increase in overall engagement and time spent on the site.” 

Facebook’s science and design expertise aim for a closed loop that feeds on, reinforces, and amplifies 
the individual user’s inclination toward fusion with the group and the tendency to over-share personal 
information. Although these vulnerabilities run deepest among the young, the tendency to over-share is not 
restricted to them. The difficulty of self-imposed discipline in the sharing of private thoughts, feelings, 
and other personal information has been amply demonstrated in social research and summarized in an 
important 2015 review by Carnegie Mellon professors Alessandro Acquisti, Laura Brandimarte, and 
George Loewenstein. They concluded that because of a range of psychological and contextual factors, 
“People are often unaware of the information they are sharing, unaware of how it can be used, and even in 
the rare situations when they have full knowledge of the consequences of sharing, uncertain about their 
own preferences....” The researchers cautioned that people are “easily influenced in what and how much 
they disclose. Moreover, what they share can be used to influence their emotions, thoughts, and behaviors. 
...” The result is alteration in “the balance of power between those holding the data and those who are the 
subjects of that data.”“4 

Facebook has Pentland’s prized God view on its side, an unparalleled resource that is drawn upon to 
remake this naturally longed-for fusion into a space of no escape. Science and capital are united in this 
long-game project. Yesterday it was the “Like” button, today it is augmented reality, and tomorrow there 
will be new innovations added to this repertoire. The company’s growth in user engagement, surplus 
capture, and revenue are evidence that these innovations have hit their marks. 

Young people crave the hive, and Facebook gives it to them, but this time it’s owned and operated by 
surveillance capital and scientifically engineered into a continuous spiral of escalating fusion, amply 
fulfilling Shaffer’s five criteria for achieving an addictive state of compulsion. Potency is engineered 
according to a recipe dictated by the hidden attributes of those who crave valorization from the group to 
fill the void where a self must eventually stand. 

These cravings may not be the sole motivations of Facebook’s currently two billion users, but they 
aptly describe the attributes upon which Facebook’s incentives are designed to bite the hardest. Climbing 
the mountain of the self-other balance is an adventure that we each must undertake: a journey of risk, 
conflict, uncertainty, and electrifying discovery. But what happens when the forces of surveillance capital 
turn the mountain into a mountain range? Look at us! Yes, you are alive! Do not look away! Why would 
you? How could you? Today, we might “like” you! 


IV. The Next Human Nature 


A growing body of evidence testifies to the psychic toll of life in the hive, where surveillance capital’s 
behavioral engineering expertise collides with the centuries-in-the-making human impulse toward self- 
construction. Researchers are already providing answers to two key questions: What are the 
psychological processes that dominate the hive? What are the individual and societal consequences of 
these processes? According to the 302 most significant quantitative research studies on the relationships 
between social media use and mental health (most of them produced since 2013), the psychological 
process that most defines the Facebook experience is what psychologists call “social comparison.” It is 
usually considered a natural and virtually automatic process that operates outside of awareness, 
“effectively forced upon the individual by his social environment” as we apply evaluative criteria tacitly 
internalized from our society, community, group, family, and friends.“ As one research review 
summarizes, “Almost at the moment of exposure, an initial holistic assessment of the similarity between 
the target and the self is made.”*2 As we go through life being exposed to other people, we naturally 
compare ourselves along the lines of similarity and contrast—I am like you. I am different from you— 
subliminal perceptions that translate into judgments—I am better than you. You are better than I. 

Researchers have come to appreciate the way in which these automatic human processes converge 
with the changing conditions of each historical era. For most of human history, people lived in small 
enclaves and were typically surrounded by others very much like themselves. Social comparisons with 
little variation are unlikely to entail great psychological risk. Research suggests that the diffusion of 
television in the second half of the twentieth century dramatically increased the intensity and negativity of 
social comparison, as it brought vivid evidence of varied and more-affluent lives dramatically different 
from one’s own. One study found an increase in criminal larceny as television diffused across society, 
awakening an awareness of and desire for consumer goods. A related issue was that increased exposure 
to television programs depicting affluence led to “the overestimation of others’ wealth and more 
dissatisfaction with one’s own life.” 

Social media marks a new era in the intensity, density, and pervasiveness of social comparison 
processes, especially for the youngest among us, who are “almost constantly online” at a time of life when 
one’s own identity, voice, and moral agency are a work in progress. In fact, the psychological tsunami of 
social comparison triggered by the social media experience is considered unprecedented. If television 
created more life dissatisfaction, what happens in the infinite spaces of social media? 

Both television and social media deprive us of real-life encounters, in which we sense the other’s 
inwardness and share something of our own, thus establishing some threads of communality. Unlike 
television, however, social media entails active self-presentation characterized by “profile inflation,” in 
which biographical information, photos, and updates are crafted to appear ever more marvelous in 
anticipation of the stakes for popularity, self-worth, and happiness.” Profile inflation triggers more 
negative self-evaluation among individuals as people compare themselves to others, which then leads to 
more profile inflation, especially among larger networks that include more “distant friends.” As one study 
concluded, “Expanding one’s social network by adding a number of distant friends through Facebook may 
be detrimental by stimulating negative emotions for users.”2 

One consequence of the new density of social comparison triggers and their negative feedback loops 
is a psychological condition known as FOMO (“fear of missing out”). It is a form of social anxiety 
defined as “the uneasy and sometimes all-consuming feeling that... your peers are doing, in the know 
about, or in possession of more or something better than you.”2! It’s a young person’s affliction that is 
associated with negative mood and low levels of life satisfaction. Research has identified FOMO with 
compulsive Facebook use: FOMO sufferers obsessively checked their Facebook feeds—during meals, 
while driving, immediately upon waking or before sleeping, and so on. This compulsive behavior is 


intended to produce relief in the form of social reassurance, but it predictably breeds more anxiety and 
more searching.°2 

Social comparison can make people do things that they might not otherwise do. Facebook’s 
experiments and Pokémon Go’s augmented reality each exploit mutual visibility and its inevitable release 
of social comparison processes for successful tuning and herding. Both of these illustrate the ways in 
which once-natural psychological processes are repurposed to heighten the effectiveness of Pentland’s 
vaunted “social pressure,” thus enabling behavior modification at scale. Social pressure is activated by “I 
want to be like you” as the risks of difference and exclusion threaten negative social comparison. 

What do we know about the mental health consequences of social comparison as it ensnares Facebook 
users, especially the young? Most of the research aimed at a deeper grasp of cause and effect in the user 
experience has been conducted with college-age participants, and even a brief review of a few key 
studies tells a grim tale, as adolescents and emerging adults run naked through these digitally mediated 
social territories in search of proof of life. A 2011 study found that social media users exposed to pictures 
of “beautiful users” developed a more negative self-image than those who were shown less attractive 
profile pictures. Men who were shown profiles of high-career-status men judged their own pursuits as 
inadequate, compared to others who saw profiles of less successful men.22 By 2013, researchers found 
that Facebook use could predict negative shifts in both how their young subjects felt moment to moment as 
well as their overall life satisfaction.4 That year, German researchers found that the “astounding... 
wealth of social information” presented on Facebook produces “a basis for social comparison and envy 
on an unprecedented scale.” Their work demonstrated that “passive following” on Facebook exacerbates 
feelings of envy and decreases life satisfaction. More than 20 percent of all recent experiences of envy 
reported by the students in the research study had been triggered by Facebook exposure. 

A three-phase investigation in 2014 found that spending a lot of time browsing profiles on Facebook 
produced a negative mood immediately afterward. Then, upon reflection, those users felt worse, 
reckoning that they had wasted their time. Instead of walking away, they typically chose to spend even 
more time browsing the network in the hope of feeling better, chasing the dream of a sudden and magical 
reversal of fortune that would justify past suffering. This cycle not only leads to more social comparison 
and more envy, but it can also predict depressive symptoms.2° 

The self-objectification associated with social comparison is also associated with other psychological 
dangers. First we present ourselves as data objects for inspection, and then we experience ourselves as 
the “it” that others see. One 2014 study demonstrated the deleterious effect of these loops on body 
consciousness. An analysis of young men and women who had used Facebook for at least six years 
concluded that, regardless of gender, more Facebook participation leads to more body surveillance. A 
sense of self-worth comes to depend on physical appearance and being perceived as a sex object. Body 
shame leads to constant rounds of manicuring self-portrayals for a largely unknown audience of 
“followers.” 

Life in the hive favors those who most naturally orient toward external cues rather than toward one’s 
own thoughts, feelings, values, and sense of personal identity.22 When considered from the vantage point 
of the self-other balance, positive social comparisons are just as pernicious as negative comparisons. 
Both are substitutes for the “hard bargain” of carving out a self that is capable of reciprocity rather than 
fusion. Whether the needle moves up or down, social comparison is the flywheel that powers the closed 
loop between the inclination toward the social mirror and its reinforcement. Both ego gratification and 
ego injury drive the chase for more external cues. 

Over time, studies increase in complexity as they try to identify the underlying mechanisms through 
which social comparison in social media is associated with symptoms of depression and feelings of 


social isolation.22 One notable three-year study published in 2017 considered both the direct Facebook 
data of more than five thousand participants as well as self-reported data on their “real-world social 
networks.” This approach enabled ongoing direct comparisons between real-world relationships and 
Facebook associations across four domains of self-reported well-being: physical health, mental health, 
life satisfaction, and body mass index. “Liking others’ content and clicking links to posts by friends,” the 
researchers summarized, “were consistently related to compromised well-being, whereas the number of 
status updates was related to reports of diminished mental health.” So strong was this relationship that “a 
1-standard-deviation increase in ‘likes clicked’... ‘links clicked’... or ‘status updates’ was associated 
with a decrease of 5%—8% of a standard deviation in self-reported mental health,” even controlling for a 
person’s initial state of well-being. The researchers’ definitive conclusion? “Facebook use does not 
promote well-being.... Individual social media users might do well to curtail their use of social media 
and focus instead on real-world relationships.” 


V. Homing to the Herd 


This is not a rehearsal. This is the show. Facebook is a prototype of instrumentarian society, not a 
prophecy. It is the first frontier of a new societal territory, and the youngest among us are its vanguard. 
The frontier experience is an epidemic of the viewpoint of the Other-One, a hyper-objectification of one’s 
own personhood shaped by the relentless amplification of life lived from the “outside looking in.” The 
consequence is a pattern of overwhelming anxiety and disorientation in the simple act of digital 
discomnection, while connection itself is haunted by fresh anxieties that paradoxically leave too many 
feeling isolated, diminished, and depressed. One wants to say that the struggles of youth can be painful in 
any era and that it is simply the destiny of today’s young people to encounter the work of self-construction 
in this milieu of digital connection and illumination, with its truly marvelous opportunities for voice, 
community, information, and exploration. One wants to say they will get through it, just as other 
generations survived the adolescent trials of their time and place. 

But this time it is not a question of simply packing their lunch and crossing our fingers as they head 
into the school-day maze of adolescent cliques, or sending them off to college knowing that they may 
stumble or fall but eventually find their passions and their people as they find themselves. This time, we 
have sent them into the raw heart of a rogue capitalism that amassed its fortune and power through 
behavioral dispossession parlayed into behavior modification in the service of others’ guaranteed 
outcomes. 

They crave the hive, just as Hall’s teenagers did in 1904, but the hive they encounter is not the 
unadulterated product of their natures and their culture of mutuality. It is a zone of asymmetrical power, 
constructed by surveillance capital as it operates in secrecy beyond confrontation or accountability. It is 
an artificial creation designed in the service of surveillance capital’s greater good. When young people 
enter this hive, they keep company with a surveillance priesthood: the world’s most-sophisticated data 
scientists, programmers, machine learning experts, and technology designers, whose single-minded 
mission to tighten the glove is mandated by the economic imperatives of surveillance capital and its “laws 
of motion.” 

Innocent hangouts and conversations are embedded in a behavioral engineering project of planetary 
scope and ambition that is institutionalized in Big Other’s architectures of ubiquitous monitoring, analysis, 
and control. In their encounter with the self-other balance, teenagers step onto a playing field already 
tilted by surveillance capital to tip them into the social mirror and keep them fixed on its reflections. 


Everything depends upon feeding the algorithms that can effectively and precisely bite on him and bite on 
her and not let go. All those outlays of genius and money are devoted to this one goal of keeping users, 
especially young users, plastered to the social mirror like bugs on the windshield. 

The research studies and first-person accounts that we have reviewed reveal the coercive underbelly 
of the instrumentarian’s much revered “confluence,” in which harmonies are achieved at the expense of 
the psychological integrity of participants. This is the world of Pentland’s “social learning,” his theory of 
“tuning” little more than the systematic manipulation of the rewards and punishments of inclusion and 
exclusion. It succeeds through the natural human inclination to avoid psychological pain. Just as ordinary 
consumers can become compulsive gamblers at the hands of the gaming industry’s behavioral 
technologies, psychologically ordinary young people are drawn into an unprecedented vortex of social 
information that automatically triggers social comparison on an equally unprecedented scale. This mental 
and emotional milieu appears to produce a virus of insecurity and anxiety that drives a young person 
deeper into this closed loop of escalating compulsion as he or she chases relief in longed-for signals of 
valorization. 

This cycle unnaturally exacerbates and intensifies the natural orientation toward the group. And 
although we all share in this disposition to varying degrees, it is most pronounced in the stages of life that 
we Call adolescence and emerging adulthood. Ethologists call this orientation “homing to the herd,” an 
adaptation of certain species, such as passenger pigeons and herring, that home to the crowd rather than to 
a particular territory. In the confrontation with human predators, however, this instinct has proven fatal. 

For example, biologist Bernd Heinrich describes the fate of the passenger pigeons, whose “social 
sense was so strong that it drew the new predator, technologically equipped humans, from afar. It made 
them not only easy targets, but easily duped.” Commercial harvesters followed the pigeons’ flight and 
nesting patterns, and then used huge nets to catch thousands of pigeons at a time, shipping millions by rail 
each year to the markets from St. Louis to Boston. The harvesters used a specific technique, designed to 
exploit the extraordinary bonds of empathy among the birds and immortalized in the term “stool pigeon.” 
A few birds would be captured first and attached to a perch with their eyes sewn shut. As these birds 
fluttered in panic, the flock would descend to “attend to them.” This made it easy for the harvesters to 
“catch and slaughter” thousands at once. The last passenger pigeon died in the Cincinnati Zoo in 1914: 
“The pigeon had no home boundaries over which to spread itself and continued to orient only to itself, so 
it could be everywhere, even to the end.... To the pigeons, the only ‘home’ they knew was in the crowd, 
and now they had become victims of it... the lack of territorial boundaries of human predators had tipped 
the scales to make their adaptation their doom.”=! 

Facebook, social media in general—these are environments engineered to induce and exaggerate this 
homing to the human herd, particularly among the young. We are lured to the social mirror, our attention 
riveted by its dark charms of social comparison, social pressure, social influence. “Online all day,” 
“online almost all day.” As we fixate on the crowd, the technologically equipped commercial harvesters 
circle quietly and cast their nets. This artificial intensification of homing to the herd can only complicate, 
delay, or impede the hard psychological bargain of the self-other balance. When we multiply this effect by 
hundreds of millions and distribute it across the globe, what might it portend for the prospects of human 
and societal development? 

Facebook is the crucible of this new dark science. It aims to perfect the relentless stimulation of social 
comparison in which natural empathy is manipulated and instrumentalized to modify behavior toward 
others’ ends. This synthetic hive is a devilish pact for a young person. In terms of sheer everyday 
effectiveness—contact, logistics, transactions, communications—turn away, and you are lost. And if you 
simply crave the fusion juice that is proof of life at a certain age and stage—turn away, and you are 


extinguished. 

It is a new phenomenon to live continuously in the milieu of the gaze of others, to be followed by 
hundreds or thousands of eyes, augmented by Big Other’s devices, sensors, beams, and waves rendering, 
recording, analyzing, and actuating. The unceasing pace, density, and volume of the gaze deliver a 
perpetual stream of evaluative metrics that raise or lower one’s social currency with each click. In China, 
these rankings are public territory, shiny badges of honor and scarlet letters that open or shut every door. 
In the West, we have “likes,” “friends,” “followers,” and hundreds of other secret rankings that invisibly 
pattern our lives. 

The extension and depth of exposure include every data point but necessarily omit the latency within 
each person, precisely because it cannot be observed and measured. This is the latency of a possible self 
that awaits ignition from that one spark caused by the caring attention of another embodied human being. It 
is in that clash of oxygen and ember that the latent is perceived, comprehended, and yanked forward into 
existence. This is real life: fleshy, soft, uncertain, and replete with silence, risk, and, when fortune smiles, 
genuine intimacy. 

Facebook entered the world bypassing old institutional boundaries, offering us freedom to connect and 
express ourselves at will. It is impossible to say what the Facebook experience might have been had the 
company chosen a path that did not depend upon surveillance revenues. Instead, we confront the sudden 
accretion of an instrumentarian power that spins our society in an unanticipated direction. Facebook’s 
applied utopistics are a prototype of an instrumentarian future, showcasing feats of behavioral engineering 
that groom populations for the rigors of instrumentarianism’s coercive harmonies. Its operations are 
designed to exploit the human inclination toward empathy, belonging, and acceptance. The system tunes 
the pitch of our behavior with the rewards and punishments of social pressure, herding the human heart 
toward confluence as a means to others’ commercial ends. 

From this vantage point, we see that the full scope of the Facebook operation constitutes a vast 
experiment in behavior modification designed not only to test the specific capabilities of its tuning 
mechanisms, as in its official “large-scale experiments,” but also to do so on the broadest possible social 
and psychological canvas. Most significantly, the applied utopistics of social pressure, its flywheel of 
social comparison, and the closed loops that bind each user to the group system vividly confirm 
Pentland’s theoretical rendering of the case. Instrumentarian social principles are evident here, not as 
hypotheses but as facts, the facts that currently constitute the spaces where our children are meant to 
“srow up.” 

What we witness here is a bet-the-farm commitment to the socialization and normalization of 
instrumentarian power for the sake of surveillance revenues. Just as Pentland stipulated, these closed 
loops are imposed outside the realm of politics and individual volition. They move in stealth, crafting 
their effects at the level of automatic psychological responses and tipping the self-other balance toward 
the pseudo-harmonies of the hive mind. In this process, the inwardness that is the necessary source of 
autonomous action and moral judgment suffers and suffocates. These are the preparatory steps toward the 
death of individuality that Pentland advocates. 

In fact, this death devours centuries of individualities: (1) the eighteenth century’s political ideal of the 
individual as the repository of inalienable dignity, rights, and obligations; (2) the early twentieth century’s 
individualized human being called into existence by history, embarking on Machado’s road because she 
must, destined to create “a life of one’s own” in a world of ever-intensifying social complexity and 
receding traditions; and (3) the late twentieth century’s psychologically autonomous individual whose 
inner resources and capacity for moral judgment rise to the challenges of self-authorship that history 
demands and act as a bulwark against the predations of power. The self-authorship toward which young 


people strive carries forward these histories, strengthening, protecting, and rejuvenating each era’s claims 
to the sanctity and sovereignty of the individual person. 

What we have seen in Facebook is a living example of the third modernity that instrumentarianism 
proffers, defined by a new collectivism owned and operated by surveillance capital. The God view 
drives the computations. The computations enable tuning. Tuning replaces private governance and public 
politics, without which individuality is merely vestigial. And just as the uncontract bypasses social 
mistrust rather than healing it, the post-political societal processes that bind the hive rely on social 
comparison and social pressure for their durability and predictive certainty, eliminating the need for trust. 
Rights to the future tense, their expression in the will to will, and their sanctification in promises are 
drawn into the fortress of surveillance capital. On the strength of that expropriation, the tuners tighten their 
grasp, and the system flourishes. 

Industrial capitalism depended upon the exploitation and control of nature, with catastrophic 
consequences that we only now recognize. Surveillance capitalism, I have suggested, depends instead 
upon the exploitation and control of human nature. The market reduces us to our behavior, transformed 
into another fictional commodity and packaged for others’ consumption. In the social principles of 
instrumentarian society, already brought to life in the experiences of our young, we can see more clearly 
how this novel capitalism aims to reshape our natures for the sake of its success. We are to be monitored 
and telestimulated like MacKay’s herds and flocks, Pentland’s beavers and bees, and Nadella’s machines. 
We are to live in the hive: a life that is naturally challenging and often painful, as any adolescent can 
attest, but the hive life in store for us is not a natural one. “Men made it.” Surveillance capitalists made it. 

The young people we have considered in this chapter are the spirits of a Christmas Yet to Come. They 
live on the frontier of a new form of power that declares the end of a human future, with its antique 
allegiances to individuals, democracy, and the human agency necessary for moral judgment. Should we 
awaken from distraction, resignation, and psychic numbing with Scrooge’s determination, it is a future that 
we may still avert. 


VI. No Exit 


When Samuel Bentham, brother of philosopher Jeremy, first designed the panopticon as a means of 
overseeing unruly serfs on the estate of Prince Potemkin in the late eighteenth century, he drew inspiration 
from the architecture of the Russian Orthodox churches that dotted the countryside. Typically, these 
structures were built around a central dome from which a portrait of an all-powerful “Christ Pantokrator” 
stared down at the congregation and, by implication, all humanity. There was to be no exit from this line 
of sight. This is the meaning of the hand and glove. The closed loop and the tight fit are meant to create the 
conditions of no exit. Once, it was no exit from God’s total knowledge and power. Today, it is no exit 
from the others, from Big Other, and from the surveillance capitalists who decide. This condition of no 
exit creeps on slippered feet. First we do not even have to look away, and later we cannot. 

In the closing lines of Jean-Paul Sartre’s existential drama No Exit, the character Garcin arrives at his 
famous realization, “Hell is other people.” This was not intended as a statement of misanthropy but rather 
a recognition that the self-other balance can never be adequately struck as long as the “others” are 
constantly “watching.” Another mid-century social psychologist, Erving Goffman, took up these themes in 
his immortal The Presentation of Self in Everyday Life. Goffman developed the idea of the “backstage” 
as the region in which the self retreats from the performative demands of social life. 

The language of backstage and onstage, inspired by observations of the theater, became a metaphor for 


the universal need for a place of retreat in which we can “be ourselves.” Backstage is where the 
“impression fostered by performance is knowingly contradicted” along with its “illusions and 
impressions.” Devices such as the telephone are “sequestered” for “private” use. Conversation is 
“relaxed,” “truthful.” It is the place where “vital secrets” can be visible. Goffman observed that in work 
as in life, “control of the backstage” allows individuals “to buffer themselves from the deterministic 
demands that surround them.” Backstage, the language is one of reciprocity, familiarity, intimacy, humor. It 
offers the seclusion in which one can surrender to the “uncomposed” face in sleep, defecation, sex, 
“whistling, chewing, nibbling, belching, and flatulence.” Perhaps most of all, it is an opportunity for 
“regression,” in which we don’t have to be “nice”: “The surest sign of backstage solidarity is to feel that 
it is safe to lapse into an asociable mood of sullen, silent irritability.” In the absence of such respite 
where a “real” self can incubate and grow, Sartre’s idea of hell begins to make sense.®2 

In a classroom of undergraduates, students discuss their strategies of self-presentation on Facebook. 
Scholars refer to these as “chilling effects”: the continuous “curation” of one’s photos, comments, and 
profile with deletions, additions, and modifications, all of it geared to the maximization of “likes” as the 
signal of one’s value in this existential marketplace. I ask if this twenty-first-century work of self- 
presentation is really that much different from what Goffman had described: have we just traded the real 
world for the virtual in constructing and performing our personas? There is a lull as the students reflect, 
and then a young woman speaks: 


The difference is that Goffman assumed a backstage where you could be your true self. For us, the backstage is shrinking. There is 
almost no place left where I can be my true self. Even when I am walking by myself, and I think I am backstage, something happens 


—an ad appears on my phone or someone takes a photo, and, I discover that I am onstage, and everything changes. 


The “everything” that changes is the sudden cognizance, part realization and part reminder, that Big 
Other knows no boundaries. Experience is seamlessly rendered across the once-reliable borders of the 
virtual and real worlds. This accrues to the immediate benefit of surveillance capital—“Welcome to 
McDonald’s!” “Buy this jacket!”—but any worldly experience can just as immediately be delivered to the 
hive: a post here, a photo there. Ubiquitous connection means that the audience is never far, and this fact 
brings all the pressures of the hive into the world and the body. 

Recent research has begun to turn to this dour fact that a team of British researchers describes as the 
“extended chilling effect.” The idea here is that people—especially, though not exclusively, young 
people—now censor and curate their real-world behavior in consideration of their own online networks 
as well as the larger prospect of the internet masses. The researchers conclude that participation in social 
media “is profoundly intertwined with the knowledge that information about our offline activities may be 
communicated online, and that the thought of displeasing ‘imagined audiences’ alters our ‘real-life’ 
behavior.” 

When I catch myself wanting to cheer the students who are anguished by connection and terrified of its 
loss, I consider the meaning of “no exit” as recounted in a personal recollection of the social psychologist 
Stanley Milgram regarding an experiment that demonstrated “the power of immediate circumstances on 
feelings and behavior.” 

Milgram’s class was studying the force with which social norms control behavior. He had the idea of 
examining the real-life phenomenon by having his students approach a person on the subway and, without 
providing any justification, simply look the person in the eye and ask for his or her seat. One afternoon, 
Milgram himself boarded the subway ready to make his contribution. Despite his years of observing and 
theorizing disturbing patterns of human behavior, it turned out that he was unprepared for his own moment 


of social confrontation. Assuming that it would be an “easy” caper, Milgram approached a passenger and 
was about to utter the “magical phrase” when “the words seemed lodged in my trachea and would simply 
not emerge. I stood there frozen, then retreated... I was overwhelmed by paralyzing inhibition.” The 
psychologist eventually hectored himself into trying again. He recounts what occurred when he finally 
approached a passenger and “choked out” his request: 


“Excuse me, sir, may I have your seat?” A moment of stark anomic panic overcame me. But the man got right up and gave me the 
seat.... Taking the man’s seat, I was overwhelmed by the need to behave in a way that would justify my request. My head sank 
between my knees, and I could feel my face blanching. I was not role-playing. I actually felt as if I were going to perish. 


Moments later the train pulled into the next station, and Milgram exited. He was surprised to discover 
that as soon as he left the train, “all the tension disappeared.” Milgram left the subway, where he vibrated 
in tune with the “others,” and that exit enabled a return to his “self.” 

Milgram identified three key themes in the subway experiment as he and his students debriefed their 
experiences. The first was a new sense of gravitas toward “the enormous inhibitory anxiety that ordinarily 
prevents us from breaching social norms.” Second was that the reactions of the “breacher” are not an 
expression of individual personality but rather are “a compelled playing out of the logic of social 
relations.” The intense “anxiety” that Milgram and others experienced in confronting a social norm “forms 
a powerful barrier that must be surmounted, whether one’s action is consequential—disobeying an 
authority—or trivial, asking for a seat on the subway.... Embarrassment and the fear of violating 
apparently trivial norms often lock us into intolerable predicaments.... These are not minor regulatory 
forces in social life, but basic ones.” 

Finally, Milgram understood that any confrontation of social norms crucially depends upon the ability 
to escape. It was not an adolescent who boarded the subway that day. Milgram was an erudite adult and 
an expert on human behavior, especially the mechanisms entailed in obedience to authority, social 
influence, and conformity. The subway was just an ordinary slice of life, not a capital-intensive 
architecture of surveillance and behavior modification, not a “personalized reward device.” Still, 
Milgram could not fight off the anxiety of the situation. The only thing that made it tolerable was the 
possibility of an exit. 

Unlike Milgram, we face an intolerable situation. Like the gamblers in their machine wombs, we are 
meant to fuse with the system and play to extinction: not the extinction of our funds but rather the 
extinction of our selves. Extinction is a design feature formalized in the conditions of no exit. The aim of 
the tuners is to contain us within “the power of immediate circumstances” as we are compelled by the 
“logic of social relations” in the hive to bow to social pressure exerted in calculated patterns that exploit 
our natural empathy. Continuously tightening feedback loops cut off the means of exit, creating impossible 
levels of anxiety that further drive the loops toward confluence. What is to be killed here is the inner 
impulse toward autonomy and the arduous, exciting elaboration of the autonomous self as a source of 
moral judgment and authority capable of asking for a subway seat or standing against rogue power. 

Inside the hive, it is easy to forget that every exit is an entrance. To exit the hive means to enter that 
territory beyond, where one finds refuge from the artificially tuned-up social pressure of the others. Exit 
leaves behind the point of view of the Other-One in favor of entering a space in which one’s gaze can 
finally settle inward. To exit means to enter the place where a self can be birthed and nurtured. History 
has a name for that kind of place: sanctuary. 


CHAPTER SEVENTEEN 


THE RIGHT TO SANCTUARY 


Refuge and prospect are opposites: refuge is small and dark; 
prospect is expansive and bright.... We need them both 
and we need them together. 


—GRANT HILDEBRAND 
“FINDING A GOOD HOME” 
ORIGINS OF ARCHITECTURAL PLEASURE 


I. Big Other Outruns Society 


That summer night when our home was destroyed by a lightning strike, we watched in the driving rain as 
the gables and rambling porches exploded in fire. Within hours a smoldering field of black ash covered 
the ground where home had been. In the months and years that followed, my recollections of the house 
took an unexpected shape, less rooms and objects than shadow, light, and fragrance. I conjured in perfect 
clarity the rush of my mother’s scent when I opened the drawer filled with her once-cherished scarves. I 
closed my eyes and saw the late-afternoon sun slicing through the velvety air by the bedroom fireplace 
with its ancient sloping mantle where our treasures were on display: a photo of my father and me, heads 
tilted toward each other, blending our two shocks of curly black hair; the miniature painted enamel boxes, 
discovered in a Parisian flea market years before the thought of motherhood, which later became the 
shelter for our children’s milk teeth huddled like secret caches of seed pearls. It was impossible to 
explain the quality of this sadness and longing: how our selves and the life of our family had evolved 
symbiotically with those spaces that we called home. How our attachments transformed a house into a 
hallowed place of love, meaning, and commemoration. 

My difficulty began to ease only when I discovered the work of Gaston Bachelard, an extraordinary 
man who had been a postal worker, physicist, philosopher, and ultimately a professor of philosophy at the 
Sorbonne: 


The old house, for those who know how to listen, is a sort of geometry of echoes. The voices of the past do not sound the same in the 


big room as in the little bed chamber.... Among the most difficult memories, well beyond any geometry that can be drawn, we must 


recapture the quality of the light; then come the sweet smells that linger in the empty rooms.... 1 


One of Bachelard’s works in particular, The Poetics of Space, is instructive as we reckon with the 
prospects of life in the no-exit shadow of Big Other and its power brokers behind the curtain. In this book 
Bachelard elaborates his notion of “topoanalysis,” the study of how our deepest relationships to inner self 
and outer world are formed in our experience of space, specifically the space we call “home”: 


The house shelters daydreaming, the house protects the dreamer, the house allows one to dream in peace.... The house is one of the 
greatest powers of integration for the thoughts, memories, and dreams of mankind.... It is body and soul. It is the human being’s first 


world. Before he is “cast into the world,”... man is laid in the cradle of the house.... Life begins well, it begins enclosed, protected, all 


warm in the bosom of the house.... = 


Home is our school of intimacy, where we first learn to be human. Its corners and nooks conceal the 
sweetness of solitude; its rooms frame our experience of relationship. Its shelter, stability, and security 
work to concentrate our unique inner sense of self, an identity that imbues our day dreams and night 
dreams forever. Its hiding places—closets, chests, drawers, locks, and keys—satisfy our need for mystery 
and independence. Doors—locked, closed, half shut, wide open—trigger our sense of wonder, safety, 
possibility, and adventure. Bachelard plumbs not only the imagery of the human house but also of nests 
and shells, the “primal images” of home that convey the absolute “primitiveness” of the need for a safe 
refuge: “Well-being takes us back to the primitiveness of the refuge. Physically, the creature endowed 
with a sense of refuge huddles up to itself, takes to cover, hides away, lies snug, concealed... a human 
being likes to ‘withdraw into his corner’... it gives him physical pleasure to do so.”2 

The shelter of home is our original way of living in space, Bachelard discovers, shaping not only the 
existential counterpoint of “home” and “away” but also many of our most fundamental ways of making 
sense of experience: house and universe, refuge and world, inside and outside, concrete and abstract, 
being and nonbeing, this and that, here and elsewhere, narrow and vast, depth and immensity, private and 
public, intimate and distant, self and other. 

Our family instinctively pursued these themes in imagining a new home. When we were finally able to 
undertake that project, we foraged for durable natural materials: old stone and scarred wooden beams that 
had weathered the storms of time. We were drawn to old furniture that had already lived many lives 
composing others’ homes. This is how the walls of the new house came to be massive, nearly a foot deep 
and packed with insulation. The result is just as we had hoped: a lush and peaceful stillness. We know 
that nothing guarantees safety and certainty in this world, but we are comforted by the serenity of this 
home and its layered silences. 

The days unfurl now within the embrace of these generous walls, where once again our spirits spread 
and root. This is how a house becomes a home and a home becomes a sanctuary. I feel this most acutely 
when I crawl into bed at night. I wait to hear my husband’s breathing in syncopation with the muffled sighs 
of our beloved dog on the floor beside us as she sprints through her ecstatic dreams. I sense beyond to the 
dense envelope of our bedroom walls and listen to their lullaby of seclusion. 

According to Big Other’s architects, these walls must come down. There can be no refuge. The primal 
yen for nests and shells is kicked aside like so much detritus from a fusty human time. With Big Other, the 
universe takes up residence in our walls, no longer the sentinels of sanctuary. Now they are simply the 
coordinates for “smart” thermostats, security cameras, speakers, and light switches that extract and render 
our experience in order to actuate our behavior. 

That our walls are dense and deep is of no importance now because the boundaries that define the 
very experience of home are to be erased. There can be no corners in which to curl up and taste the 
pleasures of solitary inwardness. There can be no secret hiding places because there can be no secrets. 


Big Other swallows refuge whole, along with the categories of understanding that originate in its 
elemental oppositions: house and universe, depth and immensity. Those ageless polarities in which we 
discover and elaborate our sense of self are casually eviscerated as immensity installs itself in my 
refrigerator, the world chatters in my toothbrush, elsewhere stands watch over my bloodstream, and the 
garden breeze stirs the chimes draped from the willow tree only to be broadcast across the planet. The 
locks? They have vanished. The doors? They are open. 

In the march of institutional interests intent on implementing Big Other, the very first citadel to fall is 
the most ancient: the principle of sanctuary. The sanctuary privilege has stood as an antidote to power 
since the beginning of the human story. Even in ancient societies where tyranny prevailed, the right of 
sanctuary stood as a fail-safe. There was an exit from totalizing power, and that exit was the entrance to a 
sanctuary in the form of a city, a community, or a temple.4 By the time of the Greeks, sanctuaries were 
sacred sites built across the ancient Greek world and consecrated to the purposes of asylum and religious 
sacrifice. The Greek word asylon means “unplunderable” and founds the notion of a sanctuary as an 
inviolable space.2 The right of asylum survived into the eighteenth century in many parts of Europe, 
attached to holy sites, churches, and monasteries. The demise of the sanctuary privilege was not a 
repudiation but rather a reflection of social evolution and the firm establishment of the rule of law. One 
historian summarized this transformation: “justice as sanctuary.”® 

In the modern era the sacredness, inviolability, and reverence that once attached to the law of asylum 
reemerged in constitutional protections and declarations of inalienable rights. English common law 
retained the idea of the castle as an inviolable fortress and translated that to the broader notion of “home,” 
a sanctuary free from arbitrary intrusion: unplunderable. The long thread of the sanctuary privilege 
reappeared in US jurisprudence. Writing in 1995, legal scholar Linda McClain argued that the equation of 
home with sanctuary has depended less on the sanctity of property rights than on a commitment to the 
“privacies of life.” As she observed, “There is a strong theme of a proper realm of inaccessibility or 
secrecy with respect to the world at large as well as a recognition of the important social dimension of 
such protected inner space....”2 

The same themes appear from the perspective of psychology. Those who would eviscerate sanctuary 
are keen to take the offensive, putting us off guard with the guilt-inducing question “What have you got to 
hide?” But as we have seen, the crucial developmental challenges of the self-other balance cannot be 
negotiated adequately without the sanctity of “disconnected” time and space for the ripening of inward 
awareness and the possibility of reflexivity: reflection on and by oneself. The real psychological truth is 
this: If you’ve got nothing to hide, you are nothing. 

One empirical study makes the point. In “Psychological Functions of Privacy,” Darhl Pedersen defines 
privacy as a “boundary control process” that invokes the decision rights associated with “restricting and 
seeking interaction.”2 Pedersen’s research identifies six categories of privacy behaviors: solitude, 
isolation, anonymity, reserve, intimacy with friends, and intimacy with family. His study shows that these 
varied behaviors accomplish a rich array of complex psychological “privacy functions” considered 
salient for psychological health and developmental success: contemplation, autonomy, rejuvenation, 
confiding, freedom, creativity, recovery, catharsis, and concealment. These are experiences without which 
we can neither flourish nor usefully contribute to our families, communities, and society. 

As the digital era intensifies and surveillance capitalism spreads, the centuries-old solution of “justice 
as sanctuary” no longer holds. Big Other outruns society and law in a self-authorized destruction of the 
right to sanctuary as it overwhelms considerations of justice with its tactical mastery of shock and awe. 
The facts of surveillance capitalism’s dominance of the division of learning, the unrepentant momentum of 
its dispossession cycle, the institutionalization of its means of behavior modification, the convergence of 


these with the requirements of social participation, and the manufacture of prediction products for trade in 
behavioral futures markets are de facto evidence of a new condition that has not been tamed by law. The 
remainder of this chapter explores the implications of this failure. What will taming require? What kind of 
life is left to us if taming fails? 


IT. Justice at the New Frontier of Power 


If sanctuary is to be preserved, synthetic declarations are required: alternative pathways that lead to a 
human future. It’s the wall, not the tunnels, that requires our attention. So far US privacy laws have failed 
to keep pace with the march of instrumentarianism. Analyses of the “invasion of privacy,” according to 
legal scholar Anita Allen, fall into “a handful of easily illustrated categories.” Allen contrasts “physical 
privacy” (sometimes called “spatial privacy’) with “informational privacy.” She observes that physical 
privacy is violated “when a person’s efforts to seclude or conceal himself or herself are frustrated.” 
Information privacy is disturbed “when data, facts, or conversations that a person wishes to secret or 
anonymize are nonetheless acquired or disclosed.”2 

In the era of Big Other, though, these categories bend and break. Physical places, including our homes, 
are increasingly saturated with informational violations as our lives are rendered as behavior and 
expropriated as surplus. In some cases we inflict this on ourselves, typically because we do not grasp the 
backstage operations and their full implications. Other violations are simply imposed upon us, as in the 
case of the talking doll, the listening TV, the hundreds of apps programmed for secret rendition, and so on. 
We have surveyed many of the objects and processes already earmarked to be smart, sensate, actuating, 
connected, and internet-enabled by surveillance capital. By the time you read these pages, there will be 
more, and more after that. It’s the sorcerer’s apprentice cursed with the perpetual filling and refilling 
driven by an unbounded claim that asserts its right to everything. 

When US scholars and jurists assess the ways in which digital capabilities challenge existing law, the 
focus is on Fourth Amendment doctrine as it circumscribes the relationship between individuals and the 
state. It is of course vital that Fourth Amendment protections catch up to the twenty-first century by 
protecting us from search and seizure of our information in ways that reflect contemporary realities of 
data production./2 The problem is that even expanded protections from the state do not shield us from the 
assault on sanctuary wrought by instrumentarian power and animated by surveillance capitalism’s 
economic imperatives. The Fourth Amendment as currently construed does not help us here. There is no 
sorcerer in sight ready to command the surveillance capitalists, in Goethe’s words, “Corner broom! Hear 
your doom.” 

Legal scholarship is just beginning to reckon with these facts. As a 2016 article on the “internet of 
things” by Fourth Amendment scholar Andrew Guthrie Ferguson concludes, “If billions of sensors filled 
with personal data fall outside of Fourth Amendment protections, a large-scale surveillance network will 
exist without constitutional limits.”4 As we have seen, it already does. Dutch scholars make a similar 
case for the inadequacy of Dutch law as it falls behind Big Other, no longer able to effectively assert the 
sanctity of the home from the invasive action of either industry or the state: “The walls no longer shield 
the individual effectively from the outside in the pursuing of... personal life without intrusion....”/2 

Many hopes today are pinned on the new body of EU regulation known as the General Data Protection 
Regulation (GDPR), which became enforceable in May 2018. The EU approach fundamentally differs 
from that of the US in that companies must justify their data activities within the GDPR’s regulatory 
framework. The regulations introduce several key new substantive and procedural features, including a 


requirement to notify people when personal data is breached, a high threshold for the definition of 
“consent” that puts limits on a company’s reliance on this tactic to approve personal data use, a 
prohibition on making personal information public by default, a requirement to use privacy by design 
when building systems, a right to erasure of data, and expanded protections against decision making 
authored by automated systems that imposes “consequential” effects on a person’s life44 The new 
regulatory framework also imposes substantial fines for violations, which will rise to a possible 4 
percent of a company’s global revenue, and it allows for class-action lawsuits in which users can 
combine to assert their rights to privacy and data protection. 

These are vital and necessary accomplishments, and the question that is most important to our story is 
whether this new regulatory regime can be a springboard to challenging the legitimacy of surveillance 
capitalism and ultimately vanquishing its instrumentarian power. In time, the world will learn if the 
GDPR can move out in front of Big Other, reasserting a division of learning aligned with the values and 
aspirations of a democratic society. Such a victory would depend upon society’s rejection of markets 
based on the dispossession of human experience as a means to the prediction and control of human 
behavior for others’ profit. 

Scholars and specialists debate the implications of the sweeping new regulations, some arguing the 
inevitability of decisive change and others arguing the likelihood of continuity over dramatic reversals of 
practice.© There are some things that we do know, however. Individuals each wrestling with the myriad 
complexities of their own data protection will be no match for surveillance capitalism’s staggering 
asymmetries of knowledge and power. If the past two decades have taught us anything, it is that the 
individual alone cannot bear the burden of this fight at the new frontier of power. 

This theme is illustrated in the odyssey of Belgian mathematician and data protection activist Paul- 
Olivier Dehaye, who in December 2016 initiated a request for his personal data collected through 
Facebook’s Custom Audiences and tracking Pixel tools, which would reveal the web pages where 
Facebook had tracked him. Dehaye probably knew more about the rogue data operations of Cambridge 
Analytica than anyone in the world, outside of its own staff and masterminds. His aim was a bottom-up 
investigative approach to uncover the secrets of its illegitimate means of political behavior modification. 

A first step was to determine what Facebook knew about him, especially the kind of data that would 
become relevant in an electoral context and thus make him, and others, vulnerable to the kinds of hidden 
maneuvers that Cambridge Analytica had employed. Dehaye wanted to understand how a citizen might 
come to ascertain the data that enabled, judging from the worldwide outrage over the revelations of secret 
online political manipulation, what many people consider a highly “consequential” threat. He carefully 
documented the twists and turns of his journey, hoping that his experience would be useful for journalists, 
citizens, and communities determined to understand the scope and political vulnerabilities of Facebook’s 
practices. Dehaye writes: 


It is of course extremely difficult to talk to a company like Facebook as an individual, so by April 2017 I had to escalate the matter to 
the Irish Data Protection Commissioner. By October 2017, after a lot of prodding, the Irish Data Protection Commissioner finally 
agreed to take the first step with my complaint, and ask Facebook for a comment. By December 2017, they had apparently received 


a response, but as of March 2018 they are still “assessing” it, despite frequent reminders. It is very hard not to see a problem here 


with respect to enforcement.t2 


In March 2018, fifteen months after his initial request, Dehaye finally received an e-mail from 
Facebook’s Privacy Operations Team. He was told that the information he sought “is not available 
through our self-service tools” but is stored in “Hive,” Facebook’s “log storage area,” where it is 


retained for “data analytics” and maintained as separate from “the data bases that power the Facebook 
site.” The company insisted that accessing the data required it to surmount “huge technical challenges.” 
“This data,” the company writes, “is also not used to directly serve the live Facebook website which 
users experience.”+8 

In our language, the information that Dehaye sought required access to the “shadow text,” specifically 
asking for, among other things, details on the targeting analyses that determined the ads he was shown on 
Facebook. The corporation’s response indicates that Hive’s data are part of this exclusive “second text,” 
in which behavioral surplus is queued up for manufacture into predictions products.!2 This process is 
completely separate from the “first text,” “which users experience.” 

Facebook makes clear that the shadow text is not available to users, despite the promotion of its self- 
service download tools that promise to give users access to their personal data retained by the company. 
Indeed, the competitive dynamics of surveillance capitalism make the shadow text a crucial proprietary 
source of advantage. Any attempt to breach its content will be experienced as an existential threat; no 
surveillance capitalist will voluntarily provide data from the shadow text. Only law can compel this 
challenge to the pathological division of learning. 

In the wake of the Cambridge Analytica scandal in March 2018, Facebook announced it would expand 
the range of personal data that it allows users to download, but even these data remain wholly contained 
within the first text, composed largely of the information that users themselves have provided, including 
information that they have deleted: friends, photos, video, ads that have been clicked, pokes, posts, 
location, and so on. These data do not include behavioral surplus, prediction products, and the fate of 
those predictions as they are used for behavioral modification, bought, and sold. When you download 
your “personal information,” you access the stage, not the backstage: the curtain, not the wizard.22 

Facebook’s response to Dehaye illustrates another consequence of the extreme asymmetries of 
knowledge at play. The company insisted that access to the requested data required it to surmount “huge 
technical challenges.” As behavioral surplus flows converge in machine-learning—based manufacturing 
operations, the sheer volume of data inputs and methods of analysis moves beyond human comprehension. 
Consider something as trivial as the case of Instagram’s machines selecting what images to show you. Its 
computations are based on varied streams of behavioral surplus from a subject user, then more streams 
from the friends in that user’s network, then more from the activities of people who follow the same 
accounts as the subject user, then the data and social links from the user’s Facebook activity. When it 
finally applies a ranking logic to predict what images the user will want to see next, that analysis also 
includes data on the past behavior of the subject user. Instagram has machines doing this “learning” 
because humans cannot.*! In the case of more “consequential” analyses, the operations are likely to be 
equally or even more complex. 

This recalls our discussion of Facebook’s “prediction engine” FBLearner Flow, where the machines 
are fed tens of thousands of data points derived from behavioral surplus, diminishing the very notion of 
the right to contest “automatic decision making.” If the algorithms are to be contestable in any meaningful 
way, it will require new countervailing authority and power, including machine resources and expertise to 
reach into the core disciplines of machine intelligence and construct new approaches that are available 
for inspection, debate, and combat. Indeed, one expert has already proposed the creation of a government 
agency—an “FDA for algorithms”—to oversee the development, distribution, sale, and use of complex 
algorithms, arguing that existing laws “will prove no match for the difficult regulatory puzzles algorithms 
pose.” 

Dehaye’s experience is but one illustration of the self-sustaining nature of a pathological division of 
learning and the insuperable burden placed on individuals moved to challenge its injustice. Dehaye is an 


activist, and his aim is not only to access data but also to document the arduousness and even absurdity of 
the undertaking. Given these realities, he suggests that the data-protection regulations are comparable to 
freedom-of-information laws. The procedures for requesting and receiving information under these laws 
are imperfect and onerous, typically undertaken by legal specialists, but nevertheless essential to 
democratic freedom. Although effective contest will require determined individuals, the individual 
alone cannot shoulder the burden of justice, any more than an individual worker in the first years of the 
twentieth century could bear the burden of fighting for fair wages and working conditions. Those 
twentieth-century challenges required collective action, and so do our own. 

In her discussion of “the life of the law,” anthropologist Laura Nader reminds us that law projects 
“possibilities of democratic empowerment” but that these are pulled forward into real life only when 
citizens actively contest injustice, using the law as a means to higher purpose. “The life of the law is the 
plaintiff,” Nader writes, a truth we saw brought to life in the action of the Spanish citizens who claimed 
the right to be forgotten. “By contesting their injustices by means of law, plaintiffs and their lawyers can 
still decide the place of law in making history.”22 These plaintiffs do not stand alone; they stand for 
citizens bonded together as the necessary means of confronting collective injustice. 

This brings us back to the GDPR and the question of its impact. The only possible answer is that 
everything will depend upon how European societies interpret the new regulatory regime in legislation 
and in the courts. It will not be the wording of the regulations but rather the popular movements on the 
ground that shape these interpretations. A century ago, workers organized for collective action and 
ultimately tipped the scales of power, and today’s “users” will have to mobilize in new ways that reflect 
our Own unique twenty-first-century “conditions of existence.” We need synthetic declarations that are 
institutionalized in new centers of democratic power, expertise, and contest that challenge today’s 
asymmetries of knowledge and power. This quality of collective action will be required if we are finally 
to replace lawlessness with laws that assert the right to sanctuary and the right to the future tense as 
essential for effective human life. 

It is already possible to see a new awakening to empowering collective action, at least in the privacy 
domain. One example is None of Your Business (NOYB), a nonprofit organization led by privacy activist 
Max Schrems. After many years of legal contest, Schrems made history in 2015 when his challenge to 
Facebook’s data-collection and data-retention practices—which he asserted were in violation of EU 
privacy law—led the Court of Justice of the European Union to invalidate the Safe Harbor agreement that 
governed data transfers between the US and the EU. In 2018 Schrems launched NOYB as a vehicle for 
“professional privacy enforcement.” The idea is to push regulators to close the gap between written 
regulations and corporate privacy practices, leveraging the threat of significant fines to change a 
company’s actual procedures. NOYB wants to become “a stable European enforcement platform” that 
unites groups of users and assists them through the litigation process while building coalitions and 
advancing “targeted and strategic litigation to maximize the impact ‘on the right to privacy.’”2° However 
this undertaking progresses, the key point for us is the way in which it points to a social void that must be 
filled with creative new forms of collective action, if the life of the law is to move against surveillance 
capitalism. 

Only time will tell if the GDPR will be a catalyst for a new phase of combat that wrangles and tames 
an illegitimate marketplace in behavioral futures, the data operations that feed it, and the instrumentarian 
society toward which they aim. In the absence of new synthetic declarations, we may be disappointed by 
the intransigence of the status quo. If the past is a prologue, then privacy, data protection, and antitrust 
laws will not be enough to interrupt surveillance capitalism. The reasons that we examined in answering 
the question “How did they get away with it?” suggest that the immense and intricate structures of 


surveillance capitalism and its imperatives will require a more direct challenge. 

This is at least one conclusion from the past decade: despite far more stringent privacy and data- 
protection laws in the EU as compared to the US, as well as a forceful commitment to antitrust, Facebook 
and Google have continued to flourish in Europe. For example, between 2010 and 2017 the compounded 
annual growth rate for Facebook daily active users was 15 percent in Europe compared to 9 percent in the 
US and Canada.~ During that period the company’s revenue grew by a compounded annual growth rate of 
50 percent in both regions.22 Between 2009 and the first quarter of 2018, Google’s share of the search 
market in Europe declined by about 2 percent while increasing in the US by about 9 percent. (Google’s 
European market share remained high, at 91.5 percent in 2018, compared to 88 percent in the US.) In the 
case of its Android mobile phones, however, Google’s market share increased by 69 percent in Europe 
compared to 44 percent in the US. Google’s Chrome browser increased its market share by 55 percent in 
Europe and 51 percent in the US.22 

Those growth rates are not mere good fortune, as our list of “how they got away with it” suggests. In 
recognition of this fact, Europe’s Data Protection Supervisor Giovanni Buttarelli told the New York Times 
that the GDPR’s impact will be determined by regulators who “will be up against well-funded teams of 
lobbyists and lawyers.”22 Indeed, corporate lawyers were already honing their strategies for the 
preservation of business as usual and setting the stage for the contests ahead. For example, a white paper 
published by one prominent international law firm rallies corporations to the barricades of data 
processing, arguing that the legal concept of “legitimate interest” offers a promising opportunity to bypass 
new regulatory obstacles: 


Legitimate interest may be the most accountable ground for processing in many contexts, as it requires an assessment and balancing 
of the risks and benefits of processing for organisations, individuals and society. The legitimate interests of the controller or a third 
party may also include other rights and freedoms. The balancing test will sometimes also include... freedom of expression, right to 


engage in economic activity, right to ensure protection of IP rights, etc. These rights must also be taken into account when balancing 


them against the individuals’ right to privacy.24 


Surveillance capitalism’s economic imperatives were already on the move in late April 2018, in 
anticipation of the GDPR taking effect that May. Earlier in April, Facebook’s CEO had announced that the 
corporation would apply the GDPR “in spirit” across the globe. In practice, however, the company was 
making changes to ensure that the GDPR would not circumscribe the majority of its operations. Until then, 
1.5 billion of its users, including those in Africa, Asia, Australia, and Latin America, were governed by 
terms of service issued by the company’s international headquarters in Ireland, meaning that these terms 
fell under the EU framework. It was in late April that Facebook quietly issued new terms of service, 
placing those 1.5 billion users under US privacy laws and thus eliminating their ability to file claims in 
Irish courts.24 


III, Every Unicorn Has a Hunter 


What life is left to us if taming fails? Without protection from surveillance capitalism and its 
instrumentarian power—their behavioral aims and societal goals—we are trapped in a condition of “no 
exit,” where the only walls are made of glass. The natural human yearning for refuge must be extinguished 
and the ancient institution of sanctuary deleted. 

“No exit” is the necessary condition for Big Other to flourish, and its flourishing is the necessary 


condition for all that is meant to follow: the tides of behavioral surplus and their transformation into 
revenue, the certainty that will meet every market player with guaranteed outcomes, the bypass of trust in 
favor of the uncontract’s radical indifference, the paradise of effortless connection that exploits the needs 
of harried second-modernity individuals and transforms their lives into the means to others’ ends, the 
plundering of the self, the extinction of autonomous moral judgment for the sake of frictionless control, the 
actuation and modification that quietly drains the will to will, the forfeit of your voice in the first person 
in favor of others’ plans, the destruction of the social relations and politics of the old and slow and still- 
unfulfilled ideals of self-determining citizens bound to the legitimate authority of democratic governance. 

Each of these exquisite unicorns has inspired the best that humanity has achieved, however imperfectly 
they have been fulfilled. But every unicorn has a hunter, and the ideals that have nurtured the liberal order 
are no exception. For the sake of this hunter, there can be no doors, no locks, no friction, no opposition 
between intimacy and distance, house and universe. There is no need for “topoanalysis” now because all 
spaces have collapsed into the one space that is Big Other. Seek not the petal-soft iridescent apex of the 
Shell. There is no purpose to curling up in its dark spire. The shell is just another connected node, and 
your daydream is already finding an audience in the pulsating net of this clamorous glass life. 

In the absence of synthetic declarations that secure the road to a human future, the intolerability of 
glass life turns us toward a societal arms race of counter-declarations in which we search for and 
embrace increasingly complex ways to hide in our own lives, seeking respite from lawless machines and 
their masters. We do this to satisfy our enduring need for sanctuary and as an act of resistance with which 
to reject the instrumentarian disciplines of the hive, its “extended chilling effects,” and Big Other’s 
relentless greed. In the context of government surveillance, the practices of “hiding” have been called 
“privacy protests” and are well-known for drawing the suspicion of law-enforcement agencies.*2 Now, 
hiding is also invoked by Big Other and its market masters, whose reach is far and deep as they install 
themselves in our walls, our bodies, and on our streets, claiming our faces, our feelings, and our fears of 
exclusion. 

I have suggested that too many of the best and brightest of a new generation devote their genius to the 
intensification of the click-stream. Equally more poignant is the way in which a new generation of 
activists, artists, and inventors feels itself called to create the art and science of hiding.“4 The intolerable 
conditions of glass life compel these young artists to dedicate their genius to the prospects of human 
invisibility, even as their creations demand that we aggressively seek and find our bearings. Their 
provocations already take many forms: signal-blocking phone cases, false fingerprint prosthetics that 
prevent your fingertips from being “used as a key to your life,” LED privacy visors to impede facial- 
recognition cameras, a quilted coat that blocks radio waves and tracking devices, a scent diffuser that 
releases a metallic fragrance when an unprotected website or network is detected on any of your devices, 
a “serendipitor app” to disrupt any surveillance “that relies on subjects maintaining predictable routines,” 
a clothing line called “Glamouflage” featuring shirts covered with representations of celebrity faces to 
confuse facial-recognition software, anti-neuroimaging surveillance headgear to obstruct digital invasion 
of brain waves, and an anti-surveillance coat that creates a shield to block invasive signals. Chicago 
artist Leo Selvaggio produces 3-D—printed resin prosthetic masks to confound facial recognition. He calls 
his effort “an organized artistic intervention.” 

Perhaps most poignant is the Backslash Tool Kit: “a series of functional devices designed for protests 
and riots of the future,” including a smart bandana for embedding hidden messages and public keys, 
independently networked wearable devices, personal black-box devices to register abuse of law 
enforcement, and fast deployment routers for off-grid communication.“° Backslash was created as part of 
a master’s thesis project at New York University, and it perfectly reflects the contest for the third 


modernity that this generation faces. The designer writes that for young, digitally native protesters, 
“connectivity is a basic human right.” Yet, he laments, “the future of technology in protests looks dark” 
because of overwhelming surveillance. His tool kit is intended to create “a space to explore and research 
the tense relationship between protests and technology and a space to cultivate dialogue about freedom of 
expression, riots and disruptive technology.” In a related development, students at the University of 
Washington have developed a prototype for “on-body transmissions with commodity devices.” The idea 
here is that readily available devices “can be used to transmit information to only wireless receivers that 
are in contact with the body,” thus creating the basis for secure and private communications independent 
of normal Wi-Fi transmissions, which can easily be detected.“ 

Take a casual stroll through the shop at the New Museum for Contemporary Art in Manhattan, and you 
pass a display of its bestseller: table-top mirrors whose reflecting surface is covered with the bright- 
orange message “Today’s Selfie Is Tomorrow’s Biometric Profile.” This “Think Privacy Selfie Mirror” 
is a project of the young Berlin-based artist Adam Harvey, whose work is aimed at the problem of 
surveillance and foiling the power of those who surveil. Harvey’s art begins with “reverse engineering... 
computer vision algorithms” in order to detect and exploit their vulnerabilities through camouflage and 
other forms of hiding. He is perhaps best known for his “Stealth Wear,” a series of wearable fashion 
pieces intended to overwhelm, confuse, and evade drone surveillance and, more broadly, facial- 
recognition software. Silver-plated fabrics reflect thermal radiation, “enabling the wearer to avert 
overhead thermal surveillance.” Harvey’s fashions are inspired by traditional Islamic dress, which 
expresses the idea that “garments can provide a separation between man and God.” Now he redirects that 
meaning to create garments that separate human experience from the powers that surveil.22 Another 
Harvey project created an aesthetic of makeup and hairstyling—blue feathers suspended from thick black 
bangs, dreadlocks that dangle below the nose, cheekbones covered in thick wedges of black and white 
paint, tresses that snake around the face and neck like octopus tentacles—all designed to thwart facial- 
recognition software and other forms of computer vision. 

Harvey is one among a growing number of artists, often young artists, who direct their work to the 
themes of surveillance and resistance. Artist Benjamin Grosser’s Facebook and Twitter “demetricators” 
are software interfaces that present each site’s pages with their metrics deleted: “The numbers of ‘likes,’ 
‘friends,’ followers, retweets... all disappear.” How is an interface that foregrounds our friend count 
changing our conceptions of friendship? he asks. “Remove the numbers and find out.” Grosser’s “Go 
Rando” project is a web browser extension that “obfuscates your feelings on Facebook” by randomly 
choosing an emoji each time you click “Like,” thus undermining the corporation’s surplus analyses as they 
compute personality and emotional profiles.22 Trevor Paglen’s richly orchestrated performance art 
combines music, photography, satellite imagery, and artificial intelligence to reveal Big Other’s 
omnipresent knowing and doing. “It’s trying to look inside the software that is running an AI... to look 
into the architectures of different computer vision systems and trying to learn what it is that they are 
seeing,” Paglen says. Chinese artist Ai Weiwei’s 2017 installation “Hansel & Gretel” created a powerful 
experience in which participants viscerally confront the surveillance implications of their own innocent 
picture taking, Instagramming, tweeting, texting, tagging, and posting. 

Our artists, like our young people, are canaries in the coal mine. That the need to make ourselves 
invisible is the theme of a brilliant artistic vanguard is another kind of message in a bottle cast from the 
front lines of mourning and revulsion. Glass life is intolerable, but so is fitting our faces with masks and 
draping our bodies in digitally resistant fabrics to thwart the ubiquitous lawless machines. Like every 
counter-declaration, hiding risks becomes an adaptation when it should be a rallying point for outrage. 
These conditions are unacceptable. Tunnels under this wall are not enough. This wall must come down. 


The greatest danger is that we come to feel at home in glass life or in the prospect of hiding from it. 
Both alternatives rob us of the life-sustaining inwardness, born in sanctuary, that finally distinguishes us 
from the machines. This is the well from which we draw the capacities to promise and to love, without 
which both the private bonds of intimacy and the public bonds of society wither and die. If we do not alter 
this course now, we leave a monumental work for the generations that follow us. Industrial capitalism 
commandeered nature only to saddle the coming generations with the burden of a burning planet. Will we 
add to this burden with surveillance capitalism’s invasion and conquest of human nature? Will we stand 
by as it subtly imposes the life of the hive while demanding the forfeit of sanctuary and the right to the 
future tense for the sake of its wealth and power? 

Paradiso calls it a revolution, and Pentland says it is the death of individuality. Nadella and Schmidt 
advocate the machine hive as our role model, with its coercive confluence and preemptive harmonies. 
Page and Zuckerberg understand the transformation of society as a means to their commercial ends. There 
are dissenters among us, to be sure, but the declaration of life without walls has thus far failed to trigger a 
mass withdrawal of agreement. This is in part the result of our dependency and in part because we do not 
yet appreciate the breadth and depth of what the architects have in store, let alone the consequences that 
this “revolution” might entail. 

Our sensibilities grow numb to the monstrosity of Big Other as its features are developed, tested, 
elaborated, and normalized. We become deaf to the lullaby of walls. Hiding from the machines and their 
masters drifts from the obsessions of the vanguard to a normal theme of social discourse and eventually 
our conversations around the dinner table. Each step down this path occurs as if in the fog of war: 
scattered fragments and incidents that appear abruptly and often in obscurity. There is little room to 
perceive the pattern, let alone its origins and meaning. Nonetheless, each deletion of the possibility of 
sanctuary leaves a void that is seamlessly and soundlessly filled by the new conditions of instrumentarian 
power. 


CHAPTER EIGHTEEN 


A COUP FROM ABOVE 


He shook with hate for things he’d never seen, 
Pined for a love abstracted from its object, 
And was oppressed as he had never been. 


—W. H. AUDEN 
SONNETS FROM CHINA, Ill 


Surveillance capitalism departs from the history of market capitalism in three startling ways. First, it 


insists on the privilege of unfettered freedom and knowledge. Second, it abandons long-standing organic 
reciprocities with people. Third, the specter of life in the hive betrays a collectivist societal vision 
sustained by radical indifference and its material expression in Big Other. In this chapter we explore each 
of these departures from historical norms and then face the question that they raise: is surveillance 
capitalism merely “capitalism”? 


I, Freedom and Knowledge 


Surveillance capitalists are no different from other capitalists in demanding freedom from any sort of 
constraint. They insist upon the “freedom to” launch every novel practice while aggressively asserting the 
necessity of their “freedom from’ law and regulation. This classic pattern reflects two bedrock 
assumptions about capitalism made by its own theorists: The first is that markets are intrinsically 
unknowable. The second is that the ignorance produced by this lack of knowledge requires wide-ranging 
freedom of action for market actors. 

The notion that ignorance and freedom are essential characteristics of capitalism is rooted in the 
conditions of life before the advent of modern systems of communication and transportation, let alone 
global digital networks, the internet, or the ubiquitous computational, sensate, actuating architectures of 
Big Other. Until the last few moments of the human story, life was necessarily local, and the “whole” was 
necessarily invisible to the “part.” 

Adam Smith’s famous metaphor of the “invisible hand” drew on these enduring realities of human life. 
Each individual, Smith reasoned, employs his capital locally in pursuit of immediate comforts and 
necessities. Each one attends to “his own security... his own gain... led by an invisible hand to promote 


an end which was no part of his intention.” That end is the efficient employ of capital in the broader 
market: the wealth of nations. The individual actions that produce efficient markets add up to a 
staggeringly complex pattern, a mystery that no one person or entity could hope to know or understand, let 
alone to direct: “The statesman, who should attempt to direct private people in what manner they ought to 
employ their capitals, would... assume an authority which could safely be trusted, not only to no single 
person, but to no council or senate whatever....”4 

The neoliberal economist Friedrich Hayek, whose work we discussed briefly in Chapter 2 as the 
foundation for the market-privileging economic policies of the past half century, drew the most basic 
tenets of his arguments from Smith’s assumptions about the whole and the part. “Adam Smith,’ Hayek 
wrote, “was the first to perceive that we have stumbled upon methods of ordering human economic 
cooperation that exceed the limits of our knowledge and perception. His ‘invisible hand’ had perhaps 
better have been described as an invisible or unsurveyable pattern.” 

As with Planck, Meyer, and Skinner, both Hayek and Smith unequivocally link freedom and ignorance. 
In Hayek’s framing, the mystery of the market is that a great many people can behave effectively while 
remaining ignorant of the whole. Individuals not only can choose freely, but they must freely choose their 
own pursuits because there is no alternative, no source of total knowledge or conscious control to guide 
them. “Human design” is impossible, Hayek says, because the relevant information flows are “beyond the 
span of the control of any one mind.” The market dynamic makes it possible for people to operate in 
ignorance without “anyone having to tell them what to do.”3 

Hayek chose the market over democracy, arguing that the market system enabled not only the division 
of labor but also “the coordinated utilization of resources based on equally divided knowledge.” This 
system, he argued, is the only one compatible with freedom. Perhaps some other kind of civilization might 
have been devised, he reckoned, “like the ‘state’ of the termite ants,” but it would not be compatible with 
human freedom.4 

Something is awry. It is true that many capitalists, including surveillance capitalists, vigorously 
employ these centuries-old justifications for their freedom when they reject regulatory, legislative, 
judicial, societal, or any other form of public interference in their methods of operation. However, Big 
Other and the steady application of instrumentarian power challenge the classic quid pro quo of freedom 
for ignorance. 

When it comes to surveillance capitalist operations, the “market” is no longer invisible, certainly not 
in the way that Smith or Hayek imagined. The competitive struggle among surveillance capitalists 
produces the compulsion toward totality. Total information tends toward certainty and the promise of 
guaranteed outcomes. These operations mean that the supply and demand of behavioral futures markets 
are rendered in infinite detail. Surveillance capitalism thus replaces mystery with certainty as it 
substitutes rendition, behavioral modification, and prediction for the old “unsurveyable pattern.” This is a 
fundamental reversal of the classic ideal of the “market” as intrinsically unknowable. 

Recall Mark Zuckerberg’s boast that Facebook would know every book, film, and song a person had 
ever consumed and that its predictive models would tell you what bar to go to when you arrive in a 
strange city, where the bartender would have your favorite drink waiting.2 As the head of Facebook’s data 
science team once reflected, “This is the first time the world has seen this scale and quality of data about 
human communication.... For the first time, we have a microscope that... lets us examine social behavior 
at a very fine level that we’ve never been able to see before....”° A top Facebook engineer put it 
succinctly: “We are trying to map out the graph of everything in the world and how it relates to each 
other.”2 

The same objectives are echoed in the other leading surveillance capitalist firms. As Google’s Eric 


Schmidt observed in 2010, “You give us more information about you, about your friends, and we can 
improve the quality of our searches. We don’t need you to type at all. We know where you are. We know 
where you’ve been. We can more or less know what you’re thinking about.”8 Satya Nadella of Microsoft 
understands all physical and institutional spaces, people, and social relationships as indexable and 
searchable: all of it subject to machine reasoning, pattern recognition, prediction, preemption, 
interruption, and modification.? 

Surveillance capitalism is not the old capitalism, and its leaders are not Smith’s or even Hayek’s 
capitalists. Under this regime, freedom and ignorance are no longer twin born, no longer two sides of the 
same coin called mystery. Surveillance capitalism is instead defined by an unprecedented convergence of 
freedom and knowledge. The degree of that convergence corresponds exactly to the scope of 
instrumentarian power. This unimpeded accumulation of power effectively hijacks the division of learning 
in society, instituting the dynamics of inclusion and exclusion upon which surveillance revenues depend. 
Surveillance capitalists claim the freedom to order knowledge, and then they leverage that knowledge 
advantage in order to protect and expand their freedom. 

Although there is nothing unusual about the prospect of capitalist enterprises seeking every kind of 
knowledge advantage in a competitive marketplace, the surveillance capitalist capabilities that translate 
ignorance into knowledge are unprecedented because they rely on the one resource that distinguishes the 
surveillance capitalists from traditional utopianists: the financial and intellectual capital that permits the 
actual transformation of the world, materialized in the continuously expanding architectures of Big Other. 
More astonishing still is that surveillance capital derives from the dispossession of human experience, 
operationalized in its unilateral and pervasive programs of rendition: our lives are scraped and sold to 
fund their freedom and our subjugation, their knowledge and our ignorance about what they know. 

This new condition unravels the neoliberal justification for the evisceration of the double movement 
and the triumph of raw capitalism: its free markets, free-market actors, and self-regulating enterprises. It 
suggests that surveillance capitalists mastered the rhetoric and political genius of the neoliberal 
ideological defense while pursuing a novel logic of accumulation that belies the most fundamental 
postulates of the capitalist worldview. It’s not just that the cards have been reshuffled; the rules of the 
game have been transformed into something that is both unprecedented and unimaginable outside the 
digital milieu and the vast resources of wealth and scientific prowess that the new applied utopianists 
bring to the table. 

We have carefully examined surveillance capitalism’s novel foundational mechanisms, economic 
imperatives, gathering power, and societal objectives. One conclusion of our investigations is that 
surveillance capitalism’s command and control of the division of learning in society are the signature 
feature that breaks with the old justifications of the invisible hand and its entitlements. The combination of 
knowledge and freedom works to accelerate the asymmetry of power between surveillance capitalists and 
the societies in which they operate. This cycle will be broken only when we acknowledge as citizens, as 
societies, and indeed as a civilization that surveillance capitalists know too much to qualify for 
freedom. 


IT. After Reciprocity 


In another decisive break with capitalism’s past, surveillance capitalists abandon the organic 
reciprocities with people that have long been a mark of capitalism’s endurance and adaptability. 
Symbolized in the twentieth century by Ford’s five-dollar day, these reciprocities hearken back to Adam 


Smith’s original insights into the productive social relations of capitalism, in which firms rely on people 
as employees and customers. Smith argued that price increases had to be balanced with wage increases 
“so that the labourer may still be able to purchase that quantity of those necessary articles which the state 
of the demand for labour... requires that he should have.”!2 The shareholder-value movement and 
globalization went a long way toward destroying this centuries-old social contract between capitalism 
and its communities, substituting formal indifference for reciprocity. Surveillance capitalism goes further. 
It not only jettisons Smith, but it also formally rescinds any remaining reciprocities with its societies. 

First, surveillance capitalists no longer rely on people as consumers. Instead, the axis of supply and 
demand orients the surveillance capitalist firm to businesses intent on anticipating the behavior of 
populations, groups, and individuals. The result, as we have seen, is that “users” are sources of raw 
material for a digital-age production process aimed at a new business customer. Where individual 
consumers continue to exist in surveillance capitalist operations—purchasing Roomba vacuum cleaners, 
dolls that spy, smart vodka bottles, or behavior-based insurance policies, just to name a few examples— 
social relations are no longer founded on mutual exchange. In these and many other instances, products 
and services are merely hosts for surveillance capitalism’s parasitic operations. 

Second, by historical standards the large surveillance capitalists employ relatively few people 
compared to their unprecedented computational resources. This pattern, in which a small, highly educated 
workforce leverages the power of a massive capital-intensive infrastructure, is called “hyperscale.” The 
historical discontinuity of the hyperscale business operation becomes apparent by comparing seven 
decades of GM employment levels and market capitalization to recent post-IPO data from Google and 
Facebook. (I have confined the comparison here to Google and Facebook because both were pure 
surveillance capitalist firms even before their public offerings.) 

From the time they went public to 2016, Google and Facebook steadily climbed to the heights of 
market capitalization, with Google reaching $532 billion by the end of 2016 and Facebook reaching $332 
billion, without Google ever employing more than 75,000 people or Facebook more than 18,000. General 
Motors took four decades to reach its highest market capitalization of $225.15 billion in 1965, when it 
employed 735,000 women and men.! Most startling is that GM employed more people during the height 
of the Great Depression than either Google or Facebook employs at their heights of market capitalization. 

The GM pattern is the iconic story of the United States in the twentieth century, before globalization, 
neoliberalism, the shareholder-value movement, and plutocracy unraveled the public corporation and the 
institutions of the double movement. Those institutions rationalized GM’s employment policies with fair 
labor practices, unionization, and collective bargaining, emblematic of stable reciprocities during the pre- 
globalization decades of the twentieth century. In the 1950s, for example, 80 percent of adults said that 
“big business” was a good thing for the country, 66 percent believed that business required little or no 
change, and 60 percent agreed that “the profits of large companies help make things better for everyone 
who buys their products or services.” 2 

Although some critics blamed these reciprocities for GM’s failure to adapt to global competition in 
the late 1980s, leading eventually to its bankruptcy in 2009, analyses have shown that chronic managerial 
complacency and doomed financial strategies bore the greatest share of responsibility for the firm’s 
legendary decline, a conclusion that is fortified by the successes of the German automobile industry in the 
twenty-first century, where strong labor institutions formally share decision making authority. 

Hyperscale firms have become emblematic of modern digital capitalism, and as capitalist inventions 
they present significant social and economic challenges, including their impact on employment and wages, 
industry concentration, and monopoly. In 2017, 24 hyperscale firms operated 320 data centers with 
anywhere between thousands and millions of servers (Google and Facebook were among the largest). 


Not all hyperscale firms are surveillance capitalists, however, and our focus here is restricted to the 
convergence of these two domains. The surveillance capitalists that operate at hyperscale or outsource to 
hyperscale operations dramatically diminish any reliance on their societies as sources of employees, and 
the few for whom they do compete, as we have seen, are drawn from the most-rarefied strata of data 
science. 

The absence of organic reciprocities with people as either sources of consumers or employees is a 
matter of exceptional importance in light of the historical relationship between market capitalism and 
democracy. In fact, the origins of democracy in both America and Britain have been traced to these very 
reciprocities. In America the violation of consumer reciprocities awakened an unstoppable march toward 
liberty as economic power translated into political power. A half century later in Britain, a grudging, 
practical, self-interested respect for the necessary interdependence of capital and labor translated into 
new patterns of political power, expressed in the gradual expansion of the franchise and the nonviolent 
shift to more-inclusive democratic institutions. Even a brief glance at these world-altering histories can 
help us grasp the degree to which surveillance capitalism diverges from capitalism’s past. 

The American Revolution is the outstanding example of how the reciprocities of consumption 
contributed to the rise of democracy. Historian T. H. Breen argues in his pathbreaking study The 
Marketplace of Revolution that it was the violation of these reciprocities that set the Revolution into 
motion, uniting disparate provincial strangers into a radical new patriotic force. Breen explains that 
American colonists had come to depend upon the “empire of goods” imported from England and that this 
dependency instilled the sense of a reciprocal social contract: “For ordinary people, the palpable 
experience of participating in an expanding Anglo-American consumer market” intensified their sense of a 
“senuine partnership” with England.1° Eventually, the British Parliament famously misjudged the rights 
and obligations of this partnership, imposing a series of taxes that turned imported goods such as cloth 
and tea into “symbols of imperial oppression.” Breen describes the originality of a political movement 
born in the shared experience of consumption, the outrage at the violation of essential producer-consumer 
interdependencies, and the determination to make “goods speak to power.” 

The translation of consumer expectations into democratic revolution occurred in three waves, 
beginning in 1765, when the Stamp Act triggered popular protests, riots, and organized resistance finally 
expressed in the “nonimportation movement” (today we would call it a consumer boycott). As Breen tells 
it, the details of the Stamp Act were less important than the colonists’ realization that England did not 
perceive them as political or economic equals bound in mutually beneficial reciprocities: “By 
compromising the Americans’ ability to purchase the goods they desired, Parliament had revealed an 
intention to treat the colonists like second-class subjects,” levying a heavy price “on the pursuit of 
material happiness.”/2 The Stamp Act was experienced as a violation of the colonists’ rights not only as 
subjects of the empire but also as consumers of the empire: it was the first translation of consumers’ 
economic power into political power, a “radically new form of politics” in which the most ordinary 
members of colonial society experienced “an exhilarating surge of empowerment.”!® Parliament 
withdrew the Stamp Act before the nonimportation movement could effectively spread across the 
colonies, and it appeared that the principle of “no taxation without representation” had prevailed. 

When the Townshend Acts were passed just two years later, in 1767, this time imposing taxes on a 
range of imported goods, a new wave of outrage mobilized people in every colony. Detailed 
nonimportation agreements turned consumer sacrifice into the front line of political resistance. The shared 
experience of violated expectations cut across regional, religious, and cultural differences, providing a 
new basis for social solidarity.2 By 1770, the Townshend Acts were also repealed, and once again it 
seemed that a full-blown rebellion would be avoided. 


The 1773 Tea Act plunged the colonies into a new phase of resistance that shifted the political focus 
from nonimportation, which depended upon merchants holding the line, to nonconsumption, which 
demanded the participation of all individuals in the unique solidarity of their shared status as 
“customers.” It was in this context that Samuel Adams proclaimed that the cause of liberty “depended on 
the ability of the American people to free themselves from ‘the Baubles of Britain.’”22 

British goods had so thoroughly come to symbolize dependency and oppression that when the tiny 
impoverished community of Harvard, Massachusetts, gathered to discuss the merchant vessels arriving in 
Boston Harbor loaded with chests of tea, they deemed it “a matter of as interesting and important a nature 
when viewed in all its Consequences not only to this Town and Province, but to America in general, and 
that for ages and generations to come, as ever came under the deliberation of this Town.” 

A year later, in 1774, the First Continental Congress convened in Philadelphia and produced a “grand 
scheme” to abolish trade with England. “It brought to fruition a brilliantly original strategy of consumer 
resistance to political oppression,” Breen writes, “one that had invited Americans to think of themselves 
as Americans even before they entertained a thought of independence.” 

In early-nineteenth-century Britain, as Daron Acemoglu and James A. Robinson have shown, the rise 
of democracy was inextricably bound to industrial capitalism’s dependency on the “masses” and their 
contribution to the prosperity necessitated by the new organization of production.*? The rise of volume 
production and its wage-earning labor force established British workers’ economic power and led to a 
growing appreciation of their political legitimacy and power. This produced a new sense of 
interdependence between ordinary people and elites. 

Acemoglu and Robinson conclude that the “dynamic positive feedback” between “inclusive economic 
institutions” (i.e., industrial firms defined by employment reciprocities) and political institutions was 
critical to Britain’s substantial and nonviolent democratic reforms. Inclusive economic institutions, they 
argue, “level the playing field,” especially when it comes to the fight for power, making it more difficult 
for elites to “crush the masses” rather than accede to their demands. Reciprocities in employment 
produced and sustained reciprocities in politics: “Clamping down on popular demands and undertaking a 
coup against inclusive political institutions would... destroy... [economic] gains, and the elites opposing 
greater democratization and greater inclusiveness might find themselves among those losing their fortunes 
from this destruction.”“ In sharp contrast to the pragmatic concessions of Britain’s early industrial 
capitalists, surveillance capitalists’ extreme structural independence from people breeds exclusion rather 
than inclusion and lays the foundation for the unique approach that we have called “radical indifference.” 


III. The New Collectivism and Its Masters of Radical Indifference 


The accumulation of freedom and knowledge combines with the lack of organic reciprocities with people 
to shape a third unusual feature of surveillance capitalism: a collectivist orientation that diverges from the 
long-standing values of market capitalism and market democracy, while also sharply departing from 
surveillance capitalism’s origins in the neoliberal worldview. For the sake of its own commercial 
success, surveillance capitalism aims us toward the hive collective. This privatized instrumentarian 
social order is a new form of collectivism in which it is the market, not the state, which concentrates both 
knowledge and freedom within its domain. 

This collectivist orientation is an unexpected development in light of surveillance capitalism’s origins 
in a neoliberal creed conceived sixty years ago as a reaction to the collectivist totalitarian nightmares of 
the mid-twentieth century. Later, with the demise of the fascist and socialist threats, neoliberal ideology 


cunningly succeeded in redefining the modern democratic state as a fresh source of collectivism to be 
resisted by any and all means. Indeed, the evisceration of the double movement has been prosecuted in the 
name of defeating the supposed collectivist hazards of “too much democracy.”=2 Now the hive emulates 
the “termite state,” which even the democracy-despising Hayek derided as incompatible with human 
freedom. 

The convergence of freedom and knowledge transforms surveillance capitalists into society’s self- 
appointed masters. From their high perch in the division of learning, a privileged priesthood of “tuners” 
rules the connected hive, cultivating it as a source of continuous raw-material supply. Just as early- 
twentieth-century managers were once taught the “administrative point of view” as the mode of 
knowledge required for the hierarchical complexities of the new large-scale corporation, today’s high 
priests practice the applied arts of radical indifference, a fundamentally asocial mode of knowledge. With 
the application of radical indifference, content is judged by its volume, range, and depth of surplus as 
measured by the “anonymous” equivalence of clicks, likes, and dwell times, despite the obvious fact that 
its profoundly dissimilar meanings originate in distinct human situations. 

Radical indifference is a response to economic imperatives, and only occasionally do we catch an 
unobstructed view of its strict application as a managerial discipline. One such occasion was a 2016 
internal Facebook memo acquired by BuzzFeed in 2018. Written by one of the company’s long-standing 
and most influential executives, Andrew Bosworth, it provided a window into radical indifference as an 
applied discipline. “We talk about the good and the bad of our work often. I want to talk about the ugly,” 
Bosworth began. He went on to explain how equivalence wins out over equality in the worldview of “an 
organism among organisms” that is essential to the march toward totality and thus the growth of 
surveillance revenues: 


We connect people. That can be good if they make it positive. Maybe someone finds love. Maybe it even saves the life of someone 
on the brink of suicide. So we connect more people. That can be bad if they make it negative. Maybe it costs a life by exposing 
someone to bullies. Maybe someone dies in a terrorist attack coordinated on our tools. And still we connect people. The ugly truth is 
that... anything that allows us to connect more people more often is de facto good. It is perhaps the only area where the metrics do 
tell the true story as far as we are concerned.... That’s why all the work we do in growth is justified. All the questionable contact 
importing practices. All the subtle language that helps people stay searchable by friends. All of the work we do to bring more 
communication in.... The best products don’t win. The ones everyone uses win... make no mistake, growth tactics are how we got 


here.2© 


As Bosworth makes clear, from the viewpoint of radical indifference the positives and negatives must 
be viewed as equivalent, despite their distinct moral meanings and human consequences. From this 
perspective the only rational objective is the pursuit of products that snare “everyone,” not “the best 
products.” 

A significant result of the systematic application of radical indifference is that the public-facing “first 
text” is vulnerable to corruption with content that would normally be perceived as repugnant: lies, 
systematic disinformation, fraud, violence, hate speech, and so on. As long as content contributes to 
“srowth tactics,” Facebook “wins.” This vulnerability can be an explosive problem on the demand side, 
the user side, but it breaks through the fortifications of radical indifference only when it threatens to 
interrupt the flow of surplus into the second “shadow” text: the one that is for them but not for us. The 
norm is that information corruption is not catalogued as problematic unless it poses an existential threat to 
supply operations—Bosworth’s imperative of connection—either because it might trigger user 
disengagement or because it might attract regulatory scrutiny. This means that any efforts toward “content 


moderation” are best understood as defensive measures, not as acts of public responsibility. 

So far, the greatest challenge to radical indifference has come from Facebook and Google’s 
overreaching ambitions to supplant professional journalism on the internet. Both corporations inserted 
themselves between publishers and their populations, subjecting journalistic “content” to the same 
categories of equivalence that dominate surveillance capitalism’s other landscapes. In a formal sense, 
professional journalism is the precise opposite of radical indifference. The journalist’s job is to produce 
news and analysis that separate truth from falsehood. This rejection of equivalence defines journalism’s 
raison d’étre as well as its organic reciprocities with its readers. Under surveillance capitalism, though, 
these reciprocities are erased. A consequential example was Facebook’s decision to standardize the 
presentation of its News Feed content so that “all news stories looked roughly the same as each other... 
whether they were investigations in The Washington Post, gossip in the New York Post, or flat-out lies in 
the Denver Guardian, an entirely bogus newspaper.”~/ This expression of equivalence without equality 
made Facebook’s first text exceptionally vulnerable to corruption from what would come to be called 
“fake news.” 

This is the context in which Facebook and Google became the focus of international attention 
following the discovery of organized political disinformation campaigns and profit-driven “fake news” 
stories during the 2016 US presidential election and the UK Brexit vote earlier that year. Economists Hunt 
Allcott and Matthew Gentzkow, who have studied these phenomena in detail, define “fake news” as 
“distorted signals uncorrelated with the truth’ that impose “private and social costs by making it more 
difficult... to infer the true state of the world....” They found that in the lead-up to the 2016 US election 
there were 760 million instances of a user reading these intentionally orchestrated lies online, or about 
three such stories for each adult American.” 

As radical indifference would predict, however, “fake news” and other forms of information 
corruption have been perennial features of Google and Facebook’s online environments. There are 
countless examples of disinformation that survived and even thrived because it fulfilled economic 
imperatives, and I point out just a few. In 2007 a prominent financial analyst worried that the subprime 
mortgage bust would harm Google’s lucrative ad business. It seems a strange observation until you learn 
that in the years prior to the Great Recession, Google eagerly welcomed shady subprime lenders into its 
behavioral futures markets, anxious to net the lion’s share of the $200 million in monthly revenue that 
mortgage lenders were spending on online advertising.22? A 2011 Consumer Watchdog report on Google’s 
advertising practices leading up to and during the Great Recession concluded that “Google has been a 
prominent beneficiary of the national home loan and foreclosure crisis... by accepting deceptive 
advertising from fraudulent operators who falsely promise unwary consumers that they can solve their 
mortgage and credit problems.” Despite these increasingly public facts, Google continued to serve its 
fraudulent business customers until 2011, when the US Treasury Department finally required the company 
to suspend advertising relationships with “more than 500 internet advertisers associated with the 85 
alleged online mortgage fraud schemes and related deceptive advertising.” 2 

Only a few months earlier, the Department of Justice had fined Google $500 million, “one of the 
largest financial forfeiture penalties in history,” for accepting ads from online Canadian pharmacies that 
encouraged Google’s US users to illegally import controlled drugs, despite repeated warnings. As the US 
Deputy Attorney General told the press, “The Department of Justice will continue to hold accountable 
companies who in their bid for profits violate federal law and put at risk the health and safety of 
American consumers.”=! 

Information corruption has also been a continuous feature of the Facebook environment. The turmoil 
associated with the 2016 US and UK political disinformation campaigns on Facebook was a well-known 


problem that had disfigured elections and social discourse in Indonesia, the Philippines, Columbia, 
Germany, Spain, Italy, Chad, Uganda, Finland, Sweden, Holland, Estonia, and the Ukraine. Scholars and 
political analysts had called attention to the harmful consequences of online disinformation for years.22 
One political analyst in the Philippines worried in 2017 that it might be too late to fix the problem: “We 
already saw the warning signs of this years ago.... Voices that were lurking in the shadows are now at the 
center of the public discourse.” 

The guiding principles of radical indifference are reflected in the operations of Facebook’s hidden 
low-wage labor force charged with limiting the perversion of the first text. Nowhere is surveillance 
capitalism’s outsized influence over the division of learning in society more concretely displayed than in 
this outcast function of “content moderation,” and nowhere is the nexus of economic imperatives and the 
division of learning more vividly exposed than in the daily banalities of these rationalized work flows 
where the world’s horrors and hate are assigned to life or death at a pace and volume that leave just 
moments to point thumbs up or down. It is only thanks to the determined reporting of a handful of 
investigative journalists and research scholars that we even have a glimpse of these highly secretive 
procedures, which now spread across a range of call centers, boutique firms, and “micro-labor” sites 
around the world. As one account notes, “Facebook and Pinterest, along with Twitter, Reddit, and 
Google, all declined to provide copies of their past or current internal moderation policy guidelines.” 

Among the few reports that have managed to assess Facebook’s operations, the theme is consistent. 
This secret workforce—some estimates reckon at least 100,000 “content moderators,” and others 
calculate the number to be much higher—operates at a distance from the corporation’s core functions, 
applying a combination of human judgment and machine learning tools.22? Sometimes referred to as 
“janitors,” they review queues of content that users have flagged as problematic. Although some general 
rules apply across the board, such as eliminating pornography and images of child abuse, a detailed 
rulebook aims to reject as little content as possible in the context of a local assessment of the minimum 
threshold of user tolerance. The larger point of the exercise is to find the point of equilibrium between the 
ability to pull users and their surplus into the site and the risk of repelling them. This is a calculation of 
radical indifference that has nothing to do with assessing the truthfulness of content or respecting 
reciprocities with users.2° This tension helps to explain why disinformation is not a priority. One 
investigative report quotes a Facebook insider: “They absolutely have the tools to shut down fake news. 
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That radical indifference produces equivalence without equality also affects the high science of 
targeted advertising. For example, journalist Julia Angwin and her colleagues at ProPublica discovered 
that Facebook “enabled advertisers to direct their pitches to the news feeds of almost 2,300 people who 
expressed interest in the topics of ‘Jew hater,’ ‘How to burn jews,’ or ‘History of why jews ruin the 
world.’”=2 As the journalists explained, “Facebook has long taken a hands-off approach to its advertising 
business.... Facebook generates its ad categories automatically based both on what users explicitly share 
with Facebook and what they implicitly convey through their online activity.” Similarly, reporters at 
BuzzFeed discovered that Google enables advertisers to target ads to people who type racist terms into 
the search bar and even suggests ad placements next to searches for “evil jew” and “Jewish control of 
banks.”2 

In the 2017 postelection environments in the United States and the United Kingdom, as “fake news” 
dominated the spotlight, journalists discovered hundreds of examples in which prediction products had 
placed ads from legitimate brands, such as Verizon, AT&T, and Walmart, alongside heinous material, 
including disinformation sites, hate speech, extreme political content, and terrorist, racist, and anti- 


Semitic publications and videos.” 


Most interesting was the assumed outrage and disbelief among surveillance capitalism’s customers: 
the advertising agencies and their clients who long ago chose to sell their souls to radical indifference, 
turning Google and Facebook into the duopoly of the online ad market and driving the massive expansion 
of surveillance capitalism.*! It had been nearly two decades since Google invented the formula that ceded 
ad placement to the equivalency metrics of click-through rates, supplanting earlier approaches that sought 
to align advertising placements with content that reflected the advertiser’s brand values. Customers 
forfeited those established reciprocities in favor of the “auto magic” of Google’s secret algorithms trained 
on proprietary behavioral surplus culled from unwitting users. Indeed, it was the radical indifference of 
click metrics that bred online displays of extremism and sensationalism in the first place, as prediction 
products favor content designed to magnetize engagement. 

The election scandals shined a harsh spotlight on these settled practices to which the world had 
already become accustomed. In the heat of controversy, many top brands made a show of suspending their 
ads on Google and Facebook until the companies eliminated corrupt content or guaranteed acceptable ad 
placements. Politicians in Europe and the United States accused Google and Facebook of profiting from 
hatred and of weakening democracy with corrupt information. Initially, both companies seemed to assume 
that the noise would quickly fade. Mark Zuckerberg said it was “crazy” to think that fake news had 
influenced the elections. Google responded to its advertising customers with vague platitudes, offering 
little in the way of change. 

This was not the first time that the leading surveillance capitalists had been called to account by 
public and press.“ In addition to the many cycles of outrage generated by Street View, Beacon, Gmail, 
Google Glass, News Feed, and other incursions, Edward Snowden’s 2013 revelations of the tech 
companies’ collusion with state intelligence agencies triggered an international eruption of loathing 
toward the surveillance capitalists. Google and Facebook learned to weather these storms with what I 
have called the “dispossession cycle,” and close observation of this new crisis suggested that a fresh 
cycle was in full throttle. As the threat of regulatory oversight grew, the adaptation phase of the cycle set 
in with a vengeance. There were public apologies, acts of contrition, attempts at mollification, and 
appearances before the US Congress and the EU Parliament.“ Zuckerberg “regretted” his “dismissive” 
attitude and prayed for forgiveness on the Jewish Day of Atonement, Yom Kippur.” Sheryl Sandberg told 
ProPublica that “we never intended or anticipated this functionality being used this way....”4° Facebook 
conceded that it could do more to combat online extremism.“ Google’s European chief told customers, 
“We apologise. Whenever anything like that happens, we don’t want it to happen and we take 
responsibility for it.” 

Consistent with the aims of the adaptation phase of the cycle, Bloomberg Businessweek observed of 
Google, “The company is trying to fight fake news without making sweeping changes.” Although both 
Google and Facebook made modest operational adjustments to try to diminish economic incentives for 
disinformation and instituted warning systems to alert users to probable corruption, Zuckerberg also used 
his super-voting power to reject a shareholder proposal that would have required the company to report 
on its management of disinformation and the societal consequences of its practices, and Google 
executives successfully fought back a similar shareholder proposal that year.22 Time would tell whether 
the companies’ users and customers would inflict financial punishment, and if so, how sustained that 
punishment might be. 

By early 2018, a quiet shift from adaptation to redirection at Facebook was already poised to 
transform crisis into opportunity. “Despite facing important challenges... we also need to keep building 
new tools to help people connect, strengthen our communities, and bring the world closer together,” 
Zuckerberg told investors.24 A Zuckerberg post followed up by a statement from the head of the 


company’s News Feed declared that henceforth the News Feed would favor posts from friends and 
family, especially posts that “spark conversations and meaningful interactions between people... we will 
predict which posts you might want to interact with your friends about.... These are posts that inspire 
back-and-forth discussion... whether that’s a post from a friend seeking advice... a news article or video 
prompting lots of discussion.... Live videos often lead to discussion among viewers... six times as many 
interactions as regular videos.” 

Radical indifference means that it doesn’t matter what is in the pipelines as long as they are full and 
flowing. Camouflaged as a retreat from corruption, the new strategy doubled down on activities rich in 
behavioral surplus, especially the live videos that Zuckerberg had long coveted. In a New York Times 
report, advertisers were quick to observe that the new rules would fuel Facebook’s “‘long-held’ video 
ambitions” and that the company had made clear its belief that its future lay in videos and video ads. One 
advertising executive commented that video content is “among the most shared and commented-upon 
content on the web.”=2 

Beyond all the explanations for the scourge of disinformation in the surveillance capitalist online 
environment is a deeper and more intransigent fact: radical indifference is a permanent invitation to the 
corruption of the first text. It sustains the pathological division of learning in society by forfeiting the 
integrity of public knowledge for the sake of the volume and scope of the shadow text. Radical 
indifference leaves a void where reciprocities once thrived. For all their freedom and knowledge, this is 
one void that surveillance capitalists will not fill because doing so would violate their own logic of 
accumulation. It is obvious that the rogue forces of disinformation grasp this fact more crisply than do 
Facebook’s or Google’s genuine users and customers as those forces learn to exploit the blind eye of 
radical indifference and escalate the perversion of learning in an open society. 


IV. What Is Surveillance Capitalism? 


Surveillance capitalism’s successful claims to freedom and knowledge, its structural independence from 
people, its collectivist ambitions, and the radical indifference that is necessitated, enabled, and sustained 
by all three now propel us toward a society in which capitalism does not function as a means to inclusive 
economic or political institutions. Instead, surveillance capitalism must be reckoned as a profoundly 
antidemocratic social force. The reasoning I employ is not mine alone. It echoes Thomas Paine’s 
unyielding defense of the democratic prospect in The Rights of Man, the polemical masterpiece in which 
he contested the defense of monarchy in Edmund Burke’s Reflections on the Revolution in France. Paine 
argued for the capabilities of the common person and against aristocratic privilege. Among his reasons to 
reject aristocratic rule was its lack of accountability to the needs of people, “because a body of men 
holding themselves accountable to nobody, ought not to be trusted by any body.”4 

Surveillance capitalism’s antidemocratic and antiegalitarian juggernaut is best described as a market- 
driven coup from above. It is not a coup d’état in the classic sense but rather a coup de gens: an 
overthrow of the people concealed as the technological Trojan horse that is Big Other. On the strength of 
its annexation of human experience, this coup achieves exclusive concentrations of knowledge and power 
that sustain privileged influence over the division of learning in society: the privatization of the central 
principle of social ordering in the twenty-first century. Like the adelantados and their silent incantations 
of the Requirimiento, surveillance capitalism operates in the declarative form and imposes the social 
relations of a premodern absolutist authority. It is a form of tyranny that feeds on people but is not of the 
people. In a surreal paradox, this coup is celebrated as “personalization,” although it defiles, ignores, 


overrides, and displaces everything about you and me that is personal. 

“Tyramy” is not a word that I choose lightly. Like the instrumentarian hive, tyranny is the obliteration 
of politics. It is founded on its own strain of radical indifference in which every person, except the tyrant, 
is understood as an organism among organisms in an equivalency of Other-Ones. Hannah Arendt observed 
that tyranny is a perversion of egalitarianism because it treats all others as equally insignificant: “The 
tyrant rules in accordance with his own will and interest... the ruler who rules one against all, and the 
‘all’ he oppresses are all equal, namely equally powerless.” Arendt notes that classical political theory 
regarded the tyrant as “out of mankind altogether... a wolf in human shape....”°2 

Surveillance capitalism rules by instrumentarian power through its materialization in Big Other, 
which, like the ancient tyrant, exists out of mankind while paradoxically assuming human shape. 
Surveillance capitalism’s tyranny does not require the despot’s whip any more than it requires 
totalitarianism’s camps and gulags. All that is needed can be found in Big Other’s reassuring messages 
and emoticons, the press of the others not in terror but in their irresistible inducements to confluence, the 
weave of your shirt saturated with sensors, the gentle voice that answers your queries, the TV that hears 
you, the house that knows you, the bed that welcomes your whispers, the book that reads you.... Big Other 
acts on behalf of an unprecedented assembly of commercial operations that must modify human behavior 
as a condition of commercial success. It replaces legitimate contract, the rule of law, politics, and social 
trust with a new form of sovereignty and its privately administered regime of reinforcements. 

Surveillance capitalism is a boundary-less form that ignores older distinctions between market and 
society, market and world, or market and person. It is a profit-seeking form in which production is 
subordinated to extraction as surveillance capitalists unilaterally claim control over human, societal, and 
political territories extending far beyond the conventional institutional terrain of the private firm or the 
market. Using Karl Polanyi’s lens, we see that surveillance capitalism annexes human experience to the 
market dynamic so that it is reborn as behavior: the fourth “fictional commodity.” Polanyi’s first three 
fictional commodities—land, labor, and money—were subjected to law. Although these laws have been 
imperfect, the institutions of labor law, environmental law, and banking law are regulatory frameworks 
intended to defend society (and nature, life, and exchange) from the worst excesses of raw capitalism’s 
destructive power. Surveillance capitalism’s expropriation of human experience has faced no such 
impediments. 

The success of this coup de gens stands as sour testimony to the thwarted needs of the second 
modernity, which enabled surveillance capitalism to flourish and still remains its richest vein for 
extraction and exploitation. In this context it is not difficult to understand why Facebook’s Mark 
Zuckerberg offers his social network as the solution to the third modernity. He envisions a totalizing 
instrumentarian order—he calls it the new global “church’—that will connect the world’s people to 
“something greater than ourselves.” It will be Facebook, he says, that will address problems that are 
civilizational in scale and scope, building “the long-term infrastructure to bring humanity together” and 
keeping people safe with “artificial intelligence” that quickly understands “what is happening across our 
community.”2° Like Pentland, Zuckerberg imagines machine intelligence that can “identify risks that 
nobody would have flagged at all, including terrorists planning attacks using private channels, people 
bullying someone too afraid to report it themselves, and other issues both local and global.”22 When 
asked about his responsibility to shareholders, Zuckerberg told CNN, “That’s why it helps to have control 
of the company.” 

For more than three centuries, industrial civilization aimed to exert control over nature for the sake of 
human betterment. Machines were our means of extending and overcoming the limits of the animal body 
so that we could accomplish this aim of domination. Only later did we begin to fathom the consequences: 


the Earth overwhelmed in peril as the delicate physical systems that once defined sea and sky gyrated out 
of control. 

Right now we are at the beginning of a new arc that I have called information civilization, and it 
repeats the same dangerous arrogance. The aim now is not to dominate nature but rather human nature. 
The focus has shifted from machines that overcome the limits of bodies to machines that modify the 
behavior of individuals, groups, and populations in the service of market objectives. This global 
installation of instrumentarian power overcomes and replaces the human inwardness that feeds the will to 
will and gives sustenance to our voices in the first person, incapacitating democracy at its roots. 

The rise of instrumentarian power is intended as a bloodless coup, of course. Instead of violence 
directed at our bodies, the instrumentarian third modernity operates more like a taming. Its solution to the 
increasingly clamorous demands for effective life pivots on the gradual elimination of chaos, uncertainty, 
conflict, abnormality, and discord in favor of predictability, automatic regularity, transparency, 
confluence, persuasion, and pacification. We are expected to cede our authority, relax our concerns, quiet 
our voices, go with the flow, and submit to the technological visionaries whose wealth and power stand 
as assurance of their superior judgment. It is assumed that we will accede to a future of less personal 
control and more powerlessness, where new sources of inequality divide and subdue, where some of us 
are subjects and many are objects, some are stimulus and many are response. 

The compulsions of this new vision threaten other delicate systems also many millennia in the making, 
but this time they are social and psychological. I am thinking here of the hard-won fruits of human 
suffering and conflict that we call the democratic prospect and the achievements of the individual as a 
source of autonomous moral judgment. Technological “inevitability” is the mantra on which we are 
trained, but it is an existential narcotic prescribed to induce resignation: a snuff dream of the spirit. 

We’ve been alerted to the “sixth extinction” as vertebrate species disappear faster than at any time 
since the end of the dinosaurs. This cataclysm is the unintended consequence of the reckless and 
opportunistic methods, also exalted as inevitable, with which industrialization imposed itself on the 
natural world because its own market forms did not hold it to account. Now the rise of instrumentarian 
power as the signature expression of surveillance capitalism augurs a different kind of extinction. This 
“seventh extinction” will not be of nature but of what has been held most precious in human nature: the 
will to will, the sanctity of the individual, the ties of intimacy, the sociality that binds us together in 
promises, and the trust they breed. The dying off of this human future will be just as unintended as any 
other. 


V. Surveillance Capitalism and Democracy 


Instrumentarian power has gathered strength outside of mankind but also outside of democracy. There can 
be no law to protect us from the unprecedented, and democratic societies, like the innocent world of the 
Tainos, are vulnerable to unprecedented power. In this way, surveillance capitalism may be viewed as 
part of an alarming global drift toward what many political scientists now view as a softening of public 
attitudes toward the necessity and inviolability of democracy itself. 

Many scholars point to a global “democratic recession” or a “deconsolidation” of Western 
democracies that were long considered impervious to antidemocratic threats.22 The extent and precise 
nature of this threat are being debated, but observers describe the bitter saudade associated with rapid 
social change and fear of the future conveyed in the lament “My children will not see the life that I 
lived.”© Such feelings of alienation and unease were expressed by many people around the world in a 


thirty-eight-nation survey published by Pew Research in late 2017. The findings suggest that the 
democratic ideal is no longer a sacred imperative, even for citizens of mature democratic societies. 
Although 78 percent of respondents say that representative democracy is “good,” 49 percent also say that 
“rule by experts” is good, 26 percent endorse “rule by a strong leader,” and 24 percent prefer “rule by the 
military.”2! 

The weakening attachment to democracy in the United States and many European countries is of 
serious concern.°* According to the Pew survey, only 40 percent of US respondents support democracy 
and simultaneously reject the alternatives. A full 46 percent find both democratic and nondemocratic 
alternatives to be acceptable, and 7 percent favor only the nondemocratic choice. The US sample trails 
Sweden, Germany, the Netherlands, Greece, and Canada in its depth of commitment to democracy, but 
other key Western democracies, including Italy, the UK, France, and Spain, along with Poland and 
Hungary, fall at or below the thirty-eight-country median of 37 percent that are exclusively committed to 
democracy. 

Many have concluded from this turmoil that market democracy is no longer viable, despite the fact that 
the combination of markets and democracy has served humanity well, helping to lift much of humankind 
from millennia of ignorance, poverty, and pain. For some of these thinkers it is the markets that must go, 
and for others it is democracy that’s slated for obsolescence. Repulsed by the social degradation and 
climate chaos produced by nearly four decades of neoliberal policy and practice, an important and varied 
group of scholars and activists argues that the era of capitalism is at end. Some propose more-humane 
economic alternatives,° some anticipate protracted decline,“ and others, repelled by social complexity, 
favor a blend of elite power and authoritarian politics in closer emulation of China’s authoritarian 
system. 

These developments alert us to a deeper truth: just as capitalism cannot be eaten raw, people cannot 
live without the felt possibility of homecoming. Hannah Arendt explored this territory more than sixty 
years ago in The Origins of Totalitarianism, where she traced the path from a thwarted individuality to a 
totalizing ideology. It was the individual’s experience of insignificance, expendability, political isolation, 
and loneliness that stoked the fires of totalitarian terror. Such ideologies, Arendt observed, appear as “a 
last support in a world where nobody is reliable and nothing can be relied upon.”®° Years later, in his 
moving 1966 essay “Education after Auschwitz,” social theorist Theodor Adorno attributed the success of 
German fascism to the ways in which the quest for effective life had become an overwhelming burden for 
too many people: “One must accept that fascism and the terror it caused are connected with the fact that 
the old established authorities... decayed and were toppled, while the people psychologically were not 
yet ready for self-determination. They proved to be unequal to the freedom that fell into their laps.”& 

Should we grow weary of our own struggle for self-determination and surrender instead to the 
seductions of Big Other, we will inadvertently trade a future of homecoming for an arid prospect of 
muted, sanitized tyranny. A third modernity that solves our problems at the price of a human future is a 
cruel perversion of capitalism and of the digital capabilities it commands. It is also an unacceptable 
affront to democracy. I repeat Thomas Piketty’s warning: “A market economy... if left to itself... contains 
powerful forces of divergence, which are potentially threatening to democratic societies and to the values 
of social justice on which they are based.”®8 This is precisely the whirlwind that we will reap at the 
hands of surveillance capitalism, an unprecedented form of raw capitalism that is surely contributing to 
the tempering of commitment to the democratic prospect as it successfully bends populations to its soft- 
spoken will. It gives so much, but it takes even more. 

Surveillance capitalism arrived on the scene with democracy already on the ropes, its early life 
sheltered and nourished by neoliberalism’s claims to freedom that set it at a distance from the lives of 


people. Surveillance capitalists quickly learned to exploit the gathering momentum aimed at hollowing 
out democracy’s meaning and muscle. Despite the democratic promise of its rhetoric and capabilities, it 
contributed to a new Gilded Age of extreme wealth inequality, as well as to once-unimaginable new 
forms of economic exclusivity and new sources of social inequality that separate the tuners from the 
tuned. Among the many insults to democracy and democratic institutions imposed by this coup des gens, I 
count the unauthorized expropriation of human experience; the hijack of the division of learning in society; 
the structural independence from people; the stealthy imposition of the hive collective; the rise of 
instrumentarian power and the radical indifference that sustains its extractive logic; the construction, 
ownership, and operation of the means of behavior modification that is Big Other; the abrogation of the 
elemental right to the future tense and the elemental right to sanctuary; the degradation of the self- 
determining individual as the fulcrum of democratic life; and the insistence on psychic numbing as the 
answer to its illegitimate quid pro quo. We can now see that surveillance capitalism takes an even more 
expansive turn toward domination than its neoliberal source code would predict, claiming its right to 
freedom and knowledge, while setting its sights on a collectivist vision that claims the totality of society. 
Though still sounding like Hayek, and even Smith, its antidemocratic collectivist ambitions reveal it as an 
insatiable child devouring its aging fathers. 

Cynicism is seductive and can blind us to the enduring fact that democracy remains our only channel 
for reformation. It is the one idea to have emerged from the long story of human oppression that insists 
upon a people’s inalienable right to rule themselves. Democracy may be under siege, but we cannot allow 
its many injuries to deflect us from allegiance to its promise. It is precisely in recognition of this dilemma 
that Piketty refuses to concede defeat, arguing that even “abnormal” dynamics of accumulation have been 
—and can again be—mitigated by democratic institutions that produce durable and effective 
countermeasures: “If we are to regain control of capital, we must bet everything on democracy...” 

Democracy is vulnerable to the unprecedented, but the strength of democratic institutions is the clock 
that determines the duration and destructiveness of that vulnerability. In a democratic society the debate 
and contest afforded by still-healthy institutions can shift the tide of public opinion against unexpected 
sources of oppression and injustice, with legislation and jurisprudence eventually to follow. 


VI. Be the Friction 


This promise of democracy reflects an enduring lesson that I absorbed from Milton Friedman at the 
University of Chicago as a nineteen-year-old undergraduate wedged in the back of a seminar room and 
straining to hear his instruction of the Chilean doctoral candidates who would soon lead their country to 
cataclysm, marching under the Friedman-Hayek flag. The professor was an optimist and a tireless 
educator who believed that legislative and judicial action invariably reflect the public opinion of twenty 
to thirty years earlier. It was an insight that he and Hayek—the two have been described as “soul mates 
and adversaries”—had crafted and transformed into systematic strategies and tactics. As Hayek told 
Robert Bork in a 1978 interview, “I’m operating on public opinion. I don’t even believe that before 
public opinion has changed, a change in the law will do any good... the primary thing is to change 
opinion....”2 Friedman’s conviction oriented him toward the long game as he threw himself into the 
distinctly nonacademic project of neoliberal evangelism with a steady stream of popular articles, books, 
and television programs. He was always sensitive to the impact of local experience, from school 
textbooks to grassroots political campaigns. 

The critical role of public opinion explains why even the most destructive “ages” do not last forever. I 


echo here what Edison said a century ago: that capitalism is “all wrong, out of gear.” The instability of 
Edison’s day threatened every promise of industrial civilization. It had to give way, he insisted, to a new 
synthesis that reunited capitalism and its populations. Edison was prophetic. Capitalism has survived the 
longue durée less because of any specific capability and more because of its plasticity. It survives and 
thrives by periodically renewing its roots in the social, finding new ways to generate new wealth by 
meeting new needs. Its evolution has been marked by a convergence of basic principles—private 
property, the profit motive, and growth—but with new forms, norms, and practices in each era.“ This is 
precisely the lesson of Ford’s discovery and the logic behind successive episodes of revitalization over 
many centuries. “There is no single variety of capitalism or organization of production,” Piketty writes. 
“This will continue to be true in the future, no doubt more than ever: New forms of organization and 
ownership remain to be invented.” Harvard philosopher Roberto Unger enlarges on this point, arguing 
that market forms can take any number of distinct legal and institutional directions, “each with dramatic 
consequences for every aspect of social life” and “immense importance for the future of humanity.”4 

When I speak to my children or an audience of young people, I try to alert them to the historically 
contingent nature of “the thing that has us” by calling attention to ordinary values and expectations before 
surveillance capitalism began its campaign of psychic numbing, “It is not OK to have to hide in your own 
life; it is not normal,” I tell them. “It is not OK to spend your lunchtime conversations comparing software 
that will camouflage you and protect you from continuous unwanted invasion.” Five trackers blocked. 
Four trackers blocked. Fifty-nine trackers blocked, facial features scrambled, voice disguised... 

I tell them that the word “search” has meant a daring existential journey, not a finger tap to already 
existing answers; that “friend” is an embodied mystery that can be forged only face-to-face and heart-to- 
heart; and that “recognition” is the glimmer of homecoming we experience in our beloved’s face, not 
“facial recognition.” I say that it is not OK to have our best instincts for connection, empathy, and 
information exploited by a draconian quid pro quo that holds these goods hostage to the pervasive strip 
search of our lives. It is not OK for every move, emotion, utterance, and desire to be catalogued, 
manipulated, and then used to surreptitiously herd us through the future tense for the sake of someone 
else’s profit. “These things are brand-new,” I tell them. “They are unprecedented. You should not take 
them for granted because they are not OK.” 

If democracy is to be replenished in the coming decades, it is up to us to rekindle the sense of outrage 
and loss over what is being taken from us. In this I do not mean only our “personal information.” What is 
at stake here is the human expectation of sovereignty over one’s own life and authorship of one’s own 
experience. What is at stake is the inward experience from which we form the will to will and the public 
spaces to act on that will. What is at stake is the dominant principle of social ordering in an information 
civilization and our rights as individuals and societies to answer the questions Who knows? Who 
decides? Who decides who decides? That surveillance capitalism has usurped so many of our rights in 
these domains is a scandalous abuse of digital capabilities and their once grand promise to democratize 
knowledge and meet our thwarted needs for effective life. Let there be a digital future, but let it be a 
human future first. 

I reject inevitability, and it is my hope that as a result of our journey together, you will too. We are at 
the beginning of this story, not the end. If we engage the oldest questions now, there is still time to take the 
reins and redirect the action toward a human future that we can call home. I turn once more to Tom Paine, 
who called upon each generation to assert its will when illegitimate forces hijack the future and we find 
ourselves hurled toward a destiny that we did not choose: “The rights of men in society are neither 
devisable, nor transferable, nor annihilable, but are descendible only; and it is not in the power of any 
generation to intercept finally and cut off the descent. If the present generation or any other, are disposed 


to be slaves, it does not lessen the right of the succeeding generation to be free: wrongs cannot have a 
legal descent.” 

Whatever has gone wrong, the responsibility to right it is renewed with each generation. Pity us and 
those who come next if we forfeit a human future to powerful companies and a rogue capitalism that fail 
to honor our needs or serve our genuine interests. Worse still would be our own voiceless capitulation to 
the message of inevitability that is power’s velvet-gloved right hand. Hannah Arendt, referring to her 
work on the origins of totalitarianism, wrote that “the natural human reaction to such conditions is one of 
anger and indignation because these conditions are against the dignity of man. If I describe these 
conditions without permitting my indignation to interfere, then I have lifted this particular phenomenon out 
of its context in human society and have thereby robbed it of part of its nature, deprived it of one of its 
important inherent qualities.” 

So it is for me and perhaps for you: the bare facts of surveillance capitalism necessarily arouse my 
indignation because they demean human dignity. The future of this narrative will depend upon the 
indignant citizens, journalists, and scholars drawn to this frontier project; indignant elected officials and 
policy makers who understand that their authority originates in the foundational values of democratic 
communities; and, especially, indignant young people who act in the knowledge that effectiveness without 
autonomy is not effective, dependency-induced compliance is no social contract, a hive with no exit can 
never be a home, experience without sanctuary is but a shadow, a life that requires hiding is no life, touch 
without feel reveals no truth, and freedom from uncertainty is no freedom. 

We return here to George Orwell, but perhaps not in the way you might imagine. In an indignant 1946 
review of James Burnham’s best seller, The Managerial Revolution, Orwell takes aim at Burnham for his 
cowardly attachment to power. The thesis of Burnham’s 1940 book was that capitalism, democracy, and 
socialism would not survive World War II. All would be replaced by a new planned centralized society 
modeled on totalitarianism. A new “managerial” class composed of executives, technicians, bureaucrats, 
and soldiers would concentrate all power and privilege in their own hands: an aristocracy of talent built 
on a semi-slave society. Throughout the book, Burnham insisted on the “inevitability” of this future and 
extolled the managerial capabilities evident in German and Russian political leadership. Writing in 1940, 
Burnham prophesied a Germany victory and the “managed” society to follow. Later, as the war still raged 
and the Red Army scored key successes, Burnham wrote a series of supplemental notes to later editions 
of the book in which he asserted with equal certainty that Russia would dominate the world. 

Orwell’s disgust is palpable: “It will be seen that at each point Burnham is predicting a continuation 
of the thing that is happening. Now, the tendency to do this is not simply a bad habit, like inaccuracy or 
exaggeration, which one can correct by taking thought. It is a major mental disease, and its roots lie partly 
in cowardice and partly in the worship of power, which is not fully separable from cowardice.” 
Burnham’s “sensational” contradictions revealed his own enthrallment with power and a complete failure 
to ascertain the creative principle in human history. “In each case,” Orwell thundered, “he was obeying 
the same instinct: the instinct to bow down before the conqueror of the moment, to accept the existing 
trend as irreversible.” 

Orwell reviled Burnham for his absolute failure of “moral effort,” expressed in his profound loss of 
bearings. Under these conditions, “literally anything can become right or wrong if the dominant class of 
the moment so wills it.” Burnham’s loss of bearings allowed him “to think of Nazism as something rather 
admirable, something that could and probably would build up a workable and durable social order.” 

Burnham’s cowardice is a cautionary tale. We are living in a moment when surveillance capitalism 
and its instrumentarian power appear to be invincible. Orwell’s courage demands that we refuse to cede 
the future to illegitimate power. He asks us to break the spell of enthrallment, helplessness, resignation, 


and numbing. We answer his call when we bend ourselves toward friction, rejecting the smooth flows of 
coercive confluence. Orwell’s courage sets us against the relentless tides of dispossession that demean all 
human experience. Friction, courage, and bearings are the resources we require to begin the shared work 
of synthetic declarations that claim the digital future as a human place, demand that digital capitalism 
operate as an inclusive force bound to the people it must serve, and defend the division of learning in 
society as a source of genuine democratic renewal. 

Arendt, like Orwell, asserts the possibility of new beginnings that do not cleave to already visible 
lines of power. She reminds us that every beginning, seen from the perspective of the framework that it 
interrupts, is a miracle. The capacity for performing such miracles is entirely human, she argues, because 
it is the source of all freedom: “What usually remains intact in the epochs of petrification and 
foreordained doom is the faculty of freedom itself, the sheer capacity to begin, which animates and 
inspires all human activities and is the hidden source... of all great and beautiful things.” 

The decades of economic injustice and immense concentrations of wealth that we call the Gilded Age 
succeeded in teaching people how they did not want to live. That knowledge empowered them to bring the 
Gilded Age to an end, wielding the armaments of progressive legislation and the New Deal. Even now, 
when we recall the lordly “barons” of the late nineteenth century, we call them “robbers.” 

Surely the Age of Surveillance Capitalism will meet the same fate as it teaches us how we do not want 
to live. It instructs us in the irreplaceable value of our greatest moral and political achievements by 
threatening to destroy them. It reminds us that shared trust is the only real protection from uncertainty. It 
demonstrates that power untamed by democracy can only lead to exile and despair. Friedman’s cycle of 
public opinion and durable law now reverts to us: it is up to us to use our knowledge, to regain our 
bearings, to stir others to do the same, and to found a new beginning. In the conquest of nature, industrial 
capitalism’s victims were mute. Those who would try to conquer human nature will find their intended 
victims full of voice, ready to name danger and defeat it. This book is intended as a contribution to that 
collective effort. 

The Berlin Wall fell for many reasons, but above all it was because the people of East Berlin said, 
“No more!” We too can be the authors of many “great and beautiful” new facts that reclaim the digital 
future as humanity’s home. No more! Let this be our declaration. 
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